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Abstract 

We present a number of results relating partial Cauchy-Littlewood sums, integrals over the compact 
classical groups, and increasing subsequences of permutations. These include: integral formulae for the 
distribution of the longest increasing subsequence of a random involution with constrained number of fixed 
points; new formulae for partial Cauchy-Littlewood sums, as well as new proofs of old formulae; relations of 
these expressions to orthogonal polynomials on the unit circle; and explicit bases for invariant spaces of the 
classical groups, together with appropriate generalizations of the straightening algorithm. 

Introduction 

Consider the following two identities: 

^sx{xi,X2,---)sx{yi,y2,---) = ^(1-2;^^^)"^ (0.1) 
lim Ec/gc/(,)ndet(l-a;,C/)-indet(l-y.C/-i)-i -TT(l-x,2/,)-^ (0.2) 

i i ij 

The first of these is the well-known identity of Cauchy ([^). The second is a formal analogue of the Szego 
limit theorem, equivalent to a theorem of [l0| . Since the right-hand sides are the same, we also have a third 
identity: 



V sx{xi,X2, . . . )sx{yi, z/2, . . . ) = lim Eueu(i) TT det(l - x,U)-^ TT det(l - y^U-Y^. (0.3) 

A i i 

Our object of study in the present work is generalizations of these three identities. 
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Our generalizations take two forms. One is to remove the limit I ^ oo. As we shall see (section H), in 



order to preserve (3^), we must then restrict the sum over partitions to involve only partitions with at most 
I parts. It is here that increasing subsequences appear: in order to rescue equations ( |0.l[ ) and (|J.2|), we must 
replace the common right-hand side with a generating function counting objects (reducing to permutations 
in an appropriate limit) without long increasing subsequences. In the case of ( |0.l[ ), the connection is via the 
Robinson-Schensted-Knuth correspondence, with its well-known connections to increasing subsequences. It 
turns out that there is also a direct connection for ( |0.2| ), in terms of the invariant theory of the unitary group. 
In particular, this gives a direct (and essentially elementary) proof of the known connection between unitary 
group integrals and increasing subsequences ( ||3^ ), as well as of the new connections given here. 

The other way in which we generalize these identities is to replace the unitary group U{1) by one of four 
other groups, including the orthogonal and symplectic groups. In terms of permutations, this corresponds to 
considering involutions (in two ways), signed permutations, and signed involutions, in addition to the original 
case of permutations; each of these conditions can be described as a particular type of symmetry condition. 



In each case, we obtain analogues of the finite / versions of equations (0.1), ( |0.2| ), and (p.3|), together with 
increasing subsequence interpretations. 
Guide to main results 

One of the classical models used to analyze increasing subsequences is the Poisson model: one generates 
a random subset of the unit square via a Poisson process, then associates a permutation to this subset in a 
canonical way (the order of the y coordinates relative to the x coordinates). The five symmetry types correspond 
to the five subgroups of Z2 x Z2 (acting on the unit square via diagonal reflections); one insists that the random 
subset be preserved by the appropriate group. It turns out that each symmetry type is naturally associated 
with a certain compact Lie group, determined as a fixed subgroup via an appropriate action of Z2 x Z2 on the 



unitary group. Our first main result (Theorem 1.2) then states that the (exact) distribution of the (length of 
the) longest increasing subsequence of a random permutation of a given symmetry type is given by the moments 
of the trace of a random element of the corresponding Lie group. 



By the Schensted correspondence, each of the five cases of Theorem |1.2| can be viewed as expressing an 
integral over one of the five groups as an appropriate sum over partitions. Each such identity specializes an 
appropriate Schur function identity (Theorem 5.2 and Corollary 5.3). For the three symmetry types with 
diagonal symmetry, this can be further generalized (essentially allowing points on the diagonal of symmetry); 
thus, for instance, if /(A) is the number of odd parts and ^(A) is the number of parts of A, then (Theorem 5.6) 

J2 a^'^^^sxix) = Eueo{i)<^ct{l + aU)Y[dct{l-XjU)-\ (0.4) 

e{x)<i j 

For a quite general class of specializations (corresponding to "super-Schur" or "hook Schur" functions), 
these Schur function sums have combinatorial interpretations in terms of increasing subsequences of multisets. 
That is, for each symmetry type and each specialization (subject to convergence conditions), we construct a 
random multiset and a notion of increasing subsequence such that (Theorem 7.1) the normalized Schur function 
sum gives the distribution of the longest increasing subsequence. These random multiset models generalize 
Johansson's 2D random growth model p3. 
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Putting these together, we obtain a connection between integrals and increasing subsequences of muhisets. 
In each case, the identity states that the dimension of a certain space of invariants is given by counting a certain 
cohection of muhisets without long increasing subsequences. For the three classical groups, we give direct proofs 
of this fact by producing an explicit basis of the invariants indexed by the appropriate multisets. Thus, for 
instance (Theorem 8.2), the centralizer algebra of the nth tensor power representation of U{1) has basis given 
by permutations of length n with no increasing subsequence of length greater than I. More generally (Theorem 
8.4), the space of simultaneous multilinear invariants of a collection of symmetric and antisymmetric, covariant 
and contravariant tensors is explicitly indexed by multisets without long increasing subsequences (generalizing 
the classical straightening algorithm). Corresponding results hold for the orthogonal (Theorems 8.5 and 8.6) 
and symplectic (Theorems 8.7 and 8.8) groups. 

The remaining collection of results is of lesser interest combinatorially, but is crucial to our asymptotic 
analysis in The key step in the analysis is to express the integrals of interest in terms of orthogonal 
polynomials on the unit circle. This is done for the classical groups in Theorem 2.3 (the remaining two groups 
reduce to the unitary group); for the unitary group the connection is immediate, while for the orthogonal group 
one must pass through orthogonal polynomials on [—1,1]. We also give a number of results indicating how 
certain modifications to the integrand affect the integral. As a consequence, we find (Corollary 4.3) that for 
each of the five natural Poisson models, the longest increasing subsequence distribution can be expressed in terms 
of the same family of orthogonal polynomials. In the sequel to this paper we determine the asymptotics of 
such polynomials via the Riemann-Hilbert method, and thus obtain the limiting longest increasing subsequence 
distribution for each of the five Poisson models as well as for the de-Poissonized versions (random symmetric 
permutations). The results are expressed in terms of the solution to the Painleve II equation, thus connecting 
to random matrix theory. Further related asymptotic work can be found in §, §, ||. 
Outline 

Section Q introduces the five symmetry types and their associated groups. In particular, we express the 
(exact) distribution of the longest increasing subsequence of a random permutation of a given symmetry type 
as an integral over the corresponding group. Section ^ expresses these integrals as determinants of Bessel 
functions related to orthogonal polynomials; this relation is given in section |^. 

Section ^ describes the extension of the integrals to include the cases when fixed points are allowed. In 
order to prove the formulae of section ^ section ^ uses representation-theoretic arguments to deduce integral 
representations of partial Cauchy-Littlewood sums, at which point the theory of symmetric functions can be 
applied. Section || briefly discusses alternate proofs of those formulae based on intermediate pfaffian forms. 

Section ^ uses a generalized Robinson-Schensted-Knuth correspondence from to relate the partial Cauchy- 
Littlewood sums to increasing subsequences of certain distributions of random multisets. 

Finally, section ^ shows that there is an intimate connection between increasing subsequences and invariants 
of the classical groups. Indeed, all of the integrals for which we give increasing subsequence interpretations 
also can be interpreted as dimensions of certain spaces of invariants. We give direct, elementary, proofs of 
these identities, by constructing bases of invariants explicitly indexed by multisets with restricted increasing 
subsequenc length. In the process, we obtain a generalization of the straightening algorithm of invariant theory, 
as well as analogues for the orthogonal and symplectic groups. We also discuss extensions to quantum groups 
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and supergroups. 

Notation We refer the reader to for notation and introduction to symmetric functions. In other notation, 
if G is a compact group, we use 

^uecfiU) (0.5) 

to denote the integral of f{U) with respect to the (normahzed) Haar measure on G. In other words, this is the 
expected value of / evaluated at a uniform random element of G. When G is the orthogonal group, we will 
occasionally need to consider the two components of G. Thus we will write 

EueoHDfiU) (0.6) 
to denote the integral of / over the coset of 0{l) of determinant ±1. In particular, we have: 

EueoHD f(U) = I Eueoii){l ± det(C/))/(C/). (0.7) 

Acknowledgments. We would like to acknowledge the following people for helpful discussions: Kurt Johansson 
for telling us about the processes generalized in section Richard Stanley for telling us about the references 
for that section, Peter Shor for spotting flaws in earlier versions of the algorithms of section |[ and Christian 
Krattenthaler for helpful comments on section |[ We would also like to thank Jeff Lagarias, Andrew Odlyzko, 
and Neil Sloane for helpful comments and enthusiasm. The work of the first author was supported in part by 
a Sloan Doctoral Foundation Fellowship. 



1 Symmetrized increasing subsequence problems 

One of the standard models used in analyzing the usual increasing subsequence problem is defined as follows. 
We say that a collection of k points in the unit square is "increasing" if for any two points {xi, yi) and {x2, 2/2), 
either xi < X2 and yi < 2/2 or xi > xi and y\ > y2- One can ask, then, for the distribution of the size of 
the largest increasing subset of n points chosen uniformly and independently from the unit square. It is not 
too difficult to see that this is the same as the distribution of the longest increasing subsequence of a random 
permutation; indeed, we can associate a (uniformly distributed) permutation to a given collection of points 
by using the relative order of the y coordinates after sorting along the x coordinates. Also of interest is the 
Poissonized analogue, in which new points are occasionally added in such a way that the number of points at 
time A is Poisson with parameter A. 

One way to generalize this model is to impose a symmetry condition on the set of points. The square has 8 
symmetries; if we insist that the symmetry preserve increasing collections, we obtain a group H of 4 elements, 
generated by the reflections through the main diagonals. Thus there are 5 possible symmetry conditions we 
can impose (including the trivial condition), which we will denote by the symbols □, 0, S, and with 
associated groups i?n, ^^0, H^, H[t], and H^. (The symbol indicates the point/linc(s) of reflection) Wc will 
also use the symbol ® to denote an arbitrary choice of the five possibilities. 
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Definition 1. We define 

® 

to be the probability that, if n points are chosen uniformly at random in the unit square, then the set S 
consisting of the images of those points under contains no increasing subset with more than I points. We 
also define a function 

Qf(A)=e-^^^. (1.1) 

The function Qf{X) corresponds to the natural Poisson model; Qf{X) is the probability that the largest 
increasing subset at time A has size at most I, and 1 — Qf{X) is the distribution function of the time at which 
an increasing subset of size I + 1 first appears. In the sequel, however, it will turn out to be convenient to use 
a different time scale; we thus define 

P®{t)=Qf{a^ty2), (1.2) 

where 00 = = 1, «□ = a^i = 2, and = 4. (Here a® is the number of Young tableaux associated to an 
element of 5® (q.v.)) 

If we map the set of points to a permutation, we obtain a permutation uniformly distributed from an 
appropriate ensemble. To be precise, define the involution t € by a; i— > n + 1 — a;. Then define an ensemble 



5® for each symmetry type as follows: 

5° = Sn (1.3) 

= {tt e S2n ■■ TT = n-\ tt{x) ^ x} (1.4) 

= {tT e S2n ■■ TT = LTT-h, 7r(x) ^ i{x)} (1.5) 

S^^ = e S2n ■■ TT = LTTl} (1.6) 

= {tT e <S'4n : TT = TT^^, TT = iTT^^t, ■k{x) ^ X, i{x)} . (I-'') 



Lemma 1.1. If a set S is chosen as above with symmetry ®, then with probability 1, the associated permutation 
is well-defined and is uniformly distributed from S® . 

This motivates the further definition 

f®i = \SXv (1-8) 

That is, /® is the number of elements of S® with no increasing subsequence of length greater than /. 
It is straightforward to compute j^®! for each case: 

\S^\=n\ (1.9) 

= = (1-10) 
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\S^\^2^n\ (1.11) 

\S^\ = (1.12) 

Note that 

Pf(t) = e-«*V2^^®1_ (1.13) 

0<n 

for □ and and 

P®(0=e-"«*V2^;®i_^ (1.14) 

for 0, S and K. 

A major reason for considering the above problems is the foUowing: 
Theorem 1.2. Fix an integer I > 0. Map H into Aut{U (21)) by 

/ <^ {U ^ {U*)~^) and {U ^ -J{U*y^J), (1.15) 

where J = i ]■ Let U®{21) be the subgroup of 11(21) fixed by the corresponding automorphisms. Then 
\h ) 

=Ec;ec/®(20(|Tr([/)P"). (1.16) 

Before giving the proof, it will be helpful to list the groups U®{21): 

U°{2l) = U{2l) (1.17) 

U^{21)^0{21) (1.18) 

U'^{21) = Sp{2l) (1.19) 

U^{2l)^U{l)xU{l) (1.20) 

U^{2l)^0{2l)nSp{2l)^U{l). (1.21) 

The last instance is the image of the fundamental representation of U{1) as a 2Z x 21 real matrix, and thus 
corresponds to the direct sum of the fundamental representation and its conjugate. 

Proof. The cases □, 0, and S are given in ]3^ ; more precisely, □ is given there as Theorem 1.1, while and 
S are given in Theorem 3.4. (Note that if vr e S^, then ttl is a fixed-point-free involution with decreasing 
subsequences corresponding to the increasing subsequence of tt.) We also give new, elementary, proofs below 



(Theorems p.2i g.Sj and 71). 

It remains to consider □ and Kl. Via the Robinson-Schensted correspondence (see section 5.1.4, for an 
excellent introduction), we can associate a pair (P, Q) of Young tableaux of the same shape to tt S S^, satisfying 
the relations — P, = Q, where S is the duality operation ("evacuation") of Schiitzenberger (ibid.). But 
there is a bijective correspondence between self-dual tableaux and domino tableaux (see, e.g., [p2[), and further 
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from domino tableaux to pairs of ordinary tableaux with disjoint content, and with shape determined only by 
the shape of the domino tableau ([0). Thus we have associated four Young tableaux (Pi, P2, Qi, Q2), where 
Pi and Qi have the same shape, P2, Q2 have the same shape. Pi and P2 have disjoint content, and Qi and Q2 
have disjoint content. This corresponds to a choice oi < m < n, independent choices of two subsets of size m 
of [1, 2, . . . n], and independent choices of tti and tt2 of length m and n — m. Furthermore, the longest increasing 
subsequence of n has length at most 21 precisely when the longest increasing subsequences of tti and 7r2 are of 
length at most Putting this together, we find 

fn(2l)^ X! (,^7) ^ril^^ri-m)l- (1-22) 



0<m<n 

On the other hand, the integral formula simplifies as follows 

\2n\ T7 . (\ \ I T^^/rr M2n 



Ec/ec/Q(2/) (I Tr([/)|2") = Eu,euii),u,euii) (I Tr([/i) + Tr(t/2)P") (1.23) 
= Eu,euiiiu,eu(i) (l(Tr(C/i) + Tr([/2))"P) (1.24) 



^Uii£U{i),U2eu{i) 



0<m<n 
i(2Z) 



(I E (^J Tr(;7i)'"Tr(C/2)"-"| ) (1.25) 

0<m<Ti ^ ^ 

= E (") /™^/°-™)^ (1-26) 



Similarly, an element tt G S"^ corresponds to a pair of Young tableaux of the same shape with disjoint 
content, and thus 

/%=P"VS- (1-28) 



On the other hand, 



(I Tr(C/)|2") = Eueuii) ((Tr(C/) + Tr(C/))^") (1.29) 
- E f!:) E.,.,) (Tr(t/)™TVM"'-'") (1.30) 



0<m<2n 



(^^")Ec/ec/w(|Tr(C/)p")- (1-31) 



□ 

It ought to be possible to give a more uniform proof of this result; the results of Section || (q.v.) may be 
relevant to this goal. 

There is an analogue of Theorem 1.2 in which 21 is replaced by 2/ + 1: 

Theorem 1.3. For any n, I > 0, 

/nVi) =Ec/ec/(2;+i)|Tr(C/)P" (1.32) 

fni2i+i)^^ueo(2i+i)\Tr{U)r (1-33) 

fn{2i+i) = Ec/GC/(0©(7(i+i) I Tr(;7)p", (1-34) 
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while 



/n(2/+l) — fn{2l) (1.35) 
/n(2i+l) = fn(2l)- (1.36) 



Also, we have the following corollary for □ and 
Corollary 1.4. For any n, I >0, 



P?i{t)^Pf'{tf (1.37) 
P^i+i{t)=PP{t)PP+i{t) (1.38) 
PfAt)=PP{t)- (1.39) 



And for 0, S, and 0, we have 
Corollary 1.5. For any n, I >0, 



pm 




(1 


piit) 




(1 


pm 


= e"* Eye(7H(2/)exp(iTr([/)). 


(1 



Proof. For 0, we have 

0<n ''■ 

But E[/go(0 Tr(C/)" = forn odd, so this is 

E[/6O(0exp(tTr([/)), (1.44) 
as required. The calculations for S and are analogous. □ 



Remark. In particular, we see that the formula ( 1.42 ) which was derived in |30(] as an expression for □ is really 
most naturally interpreted in H terms. 

As an aside, we observe that if we remove the condition that the symmetries under consideration preserve 
increasing sets, but insist that the corresponding sets should still give permutations, there is one further type 
of symmetry allowed, namely rotation by 90 degrees. In terms of permutations, this is the set 

5° = {tt e Sin -.TT^^i} \S°J - (2n)!/n! (1.45) 

Such permutations correspond to pairs of tableaux (P, Q*) with n elements such that P and Q have the same 
shape and disjoint content. It follows that the length I of the longest increasing subsequence from this set has 
the same distribution as max(2^+(7r), 21^ (t:) — 1), where tt is randomly chosen from Sn, ^^(tt) is the increasing 
subsequence length of tt, and /^(tt) is the decreasing subsequence length of tt. In particular, the bound of Erdos 
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and Szekeres implies that /°; = for n > I , and thus no integral formula a la Theorem 1.2 can exist for this 
case. There is a determinant formula, however, which can be obtained from the following symmetric function 
identity: 



E 

£(A)<i + 

i{\')<r 



One can either derive this via the approach in , or simply use the Jacobi-Trudi identity together with matrix 
manipulation as in [l3| . This then gives the following formulae: 

E ^"(20 7^ = det (Vri^ ^^-^^^ 

0<n \( j0<»<i,0<j<2i/ 



ro _ A >-( (l7W)o<''<;,0<j<2i + l 

Z./n(2;+i)-^ - dot I 



^ I "(I+iirjyr J < i < i + 1 , < j < 2 1 + 



(1.48) 



It is not clear how to obtain asymptotic information from the formulae, however, so we will not discuss this 
case further. (The techniques of |^ may be applicable, however.) 

2 Determinantal formulae 



It turns out that each of the formulae of Corollary 1.5 can be expressed in terms of a Toeplitz or Hankel 



determinant, related to orthogonal polynomials on the miit circle. Indeed, this correspondence is more general. 
For the unitary group: 

Theorem 2.1. Let f{z), g{z) he any Junctions on the unit circle. Then 

Eu^u(i)det{f{U))detig{U^))^det(^f f{e^O)g{e-^0)e^^^-''^Ode] . (2.1) 



\ ^ ■ J / 0<j,k<l 

Proof. Using the Weyl integration formula for the unitary group, we have 

Eaec/(0det(/(f7))det(5(C/t))^-l- / W'' - e'''\' U f{e''^)9{e~''')d9,. (2.2) 



The result follows from the standard theory of Toeplitz determinants, or by the classic formula for the 
integral of a product of two (generalized) Vandermonde determinants (see, for instance, ||^): 

i / dei{(i)j{xk))Q<j.k<idet{%l)j{xk))o<j.k<i\\ iJ'idxj) = det ( / (t)j{x)i}k{x)^{dx)\ , (2.3) 

JS' j \Js ) 0<3,k<l 

for any measure fi on any set 5". □ 
Similarly, for the orthogonal and symplectic groups, we have: 
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Theorem 2.2. Let g{z) be any function on the unit circle such that the integrals 

g{e^')g{e-^')e^^U9 (2.4) 



1 

2^ 



[0,27r] 



are well-defined. Then 



Ec/eo+(20 det(5([/)) ^ det(ij_fc + tj+fe)o<j,fc<i (2.5) 

Et/eo-(2/) det(5([/)) .g(l).g(-l) det(tj_fe - tj+fc+2)o<j,fe<i-i (2.6) 

Ec/GO+(2;+i) det(.g(C/)) = .g(l) det(tj_fc - ij+fc+i)o<j,fc<i (2-7) 

E[/eo-(2/+i) det(g(C/)) ^ g{-l) det{Lj^k + Lj+k+i)o<j.k<i (2.8) 

E(7eSp(2/) det{g{U)) dct(tj_fc - ij+/c+2)o<j,/c<i, (2.9) 

except that ^ueo+io) det(.g(C/)) = 1. 

Proof. As observed in Proposition 3.1 of |2^, integrals over the orthogonal and symplectic groups can be 
expressed as Hankel determinants; thus, for instance, 

Ec/eo+(20 det(5([/)) cx det ( i / g(e^^).g(e-*^)p,(cos(0))pfc(cos(0))d0 ) (2.10) 

for any polynomials pj with deg{pj) = j. In particular, this must be true when pj(cos(6')) = cos{j6) (Chebyshev 
polynomials). In that case, noting that ik ~ I'-k, the jk coefficient of the determinant is 

- / g{e''')g{e-''')cos{je)cos{k9)d0 = Lj^k + h+k- (2.11) 

I" J[0,27r] 

The constant of proportionality can be determined by comparing the two sides when g — 1, and thus tj = Sjo- 
For the other cases, we take pj{coa{9)) to be 

Mij + m sin((j + l/2)g) cos((j + 1/2)9) 

sm{e) ' sin(0/2) ' cos(0/2) ' ^ ' ' 

as appropriate. □ 

For our purposes, we will need the following related result: 
Theorem 2.3. Let g(z) be as above, and consider the (symmetric) inner product on polynomials 



(p(z),g(z)) = ^ / pie^')qie~^')gie^<')g{e-^0)d9. (2.13) 



Let TTjiz) be the monic orthogonal polynomials on the unit circle relative to that inner product, and define 
Nj = {'Kj(z),TTj{z)) . Lf the polynomials 7Tj{z) are well-defined, then 

Ec/ec/(oldet(g([/))P= [] N, (2.14) 

0<i</ 

E[/GO+(o)det(5([/)) = l (2.15) 
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E(760+(2i) det(5(C/)) 


o<i</-i 


2(1 - 


f 7r2j+2(0)) 


(2.16) 


E;7GO-(20 det(.g(C/)) 


-.9(i),9(-i) n 

o<j<; 


N 

-1 


2,+2(l-^2j+2(0))-l 


(2.17) 


Ec/60+(2i+i) det(5(C/)) 


-5(1) n 

0<]<l 


(1- 




(2.18) 


Ec/GO-(2i+i)det(.g(C/)) 


=.9(-i) n ^2,- 

0<i</ 




+ ^2, + l(0))-^ 


(2.19) 


Ec/GSp(20 det(5(C/)) 


= n ^2,+2(i- 


7r2j4 


-2(0))-^ 


(2.20) 



0<j<l 

The proof will be given in Section |^ below. By combining these formulae, we obtain: 
Corollary 2.4. Let g{z) be as above. Then for any I > 0, 

g{l)g{-l) EueuH) I det{g{U))\^ - (Ec;eo+(i+i) detig{U)))CEueO- (i+i) det(g((7))). (2.21) 
Proof. Use Ni = (1 - 7r;(0)2)iV;_i which follows from ( |3.1l|) below. □ 

In our case, the function g{z) = e*^, and thus everything is related to the orthogonal polynomials on the 
circle for the weight function exp(t(z + 1/^)) = exp(2icos0). Let these polynomials be TTi(z;t) with norms 
Ni{t), and define five functions Di{t), D^^{t) by 



A(i) = det(/j_fe(2t))o<j;fc<z 

= n ^^-w 

0<j<l 

D^-{t) = i det(/,_fe(2t) + Ij+k{2t))o<j,k<i 

= No{t) H N2j+2{t){l + 7r2j+2{0;t))-' 
o<i</-i 

D++{t) = det(/,_fe(2<) - Ij+k+2{2t))o<jM<i 
= n A^2,+2(t)(l-7r2,+2(0;t))-i 

0<j<l 

D+-{t) ^ det(/j_fe(2i) - /j+fe+i(2i))o<j^fc<i 
= n ^2,+i(<)(l-7r2,+i(0;t))-i 

0<j</ 

i?;-+(t) - det(/,_fe(2t) + Ij+k+i{2t))o<j,k<i 
= n ^2,+i(t)(l + ^2j+i(0;t))-\ 

0<j</ 



2.22) 
2.23) 

2.24) 
2.25) 
2.26) 

2.27) 
2.28) 

2.29) 
2.30) 

2.31) 
2.32) 



where 



lJ2t) = / e2*-^«e^^« = y ^-^ (2.33) 
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is a modified Bessel function of the first kind. 
Then 



E[/e[/(0 |exp(tTr(C/))|2 = Di{t) (2.34) 

E[/eo+(20 exp(tTr((7)) = Di-{t) (2.35) 

Et/GO-(20 exp(iTr(t/)) - i^++i(i) (2.36) 

Ec/eo+(2i+i) exp(tTr(;7)) = e*i^,+-(t) (2.37) 

E[/eo-(2i+i) exp(tTr((7)) = e-*i?r+W- (2-38) 



In summary: 

Theorem 2.5. for Z > 0, we have the following formulae: 



pFit) 


= e' 






pm 


= e' 


[D--{t) + Dl 


\it)] /2 


pH+iit) 


— e' 


[e'D+-{t)+e 


-'D-+{t)] /2 


piit) 


= e' 






P^ii) 


— e' 






pi+iit) 


— e' 


Di{t)Di+iit) 




Pfiii) 


= e' 







(2.39) 
(2.40) 
(2.41) 
(2.42) 
(2.43) 
(2.44) 
(2.45) 



except that P^{t) = e-*^/'^Do~{t) = e-*'/^ 



In H], wc will also need the following limits, which follow immediately from Szego's limit theorem and the 
analogue for orthogonal polynomials on a finite interval: 

Theorem 2.6. For any real t > 0, 

(2.46) 
(2.47) 
(2.48) 
(2.49) 
(2.50) 

Remark. These limits are also valid as limits of formal power series in t; see ||l^ and Theorem B.IO below. 



lim Di{t) 

I — >-co 




lim D--{t) 

— >-oo 




lim D++{t) 




lim D+-{t) 




lim D^+{t) 
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Corollary 2.7. For any real t > 0, 

e-''Di(t) = \{N,{t)-^ (2.51) 



3>l 

'''^D--{t) = []^2,+2W"'(l + ^2,+2(0;t)) (2.52) 



e 

3>l 



e 



'^D++{t) = []iV2,+2(0"'(l-^2,+2(0;t)) (2.53) 



= J]iV2,.+,(i)-i(l-7r2,+i(0;t)) (2.54) 
^_,V2-t^-+(i) ^ J]iV2,+i(i)-'(l + ^2,+i(0;t)). (2.55) 

3 Orthogonal polynomial identities 

Let z/;(x) on [—1, 1] and f{z) on the unit circle be related by 

f{e'%^w{cose). (3.1) 

Associated to / are five sets of monic orthogonal polynomials: 7r;(z) with respect to the (symmetric) inner 
product 



{p{z),q{z))^^ [ p{e^%{e-^')f{e^')de. (3.2) 

'[0,27r] 



2tt 

±± 



on the unit circle, and the four sets (z), with respect to the inner products 

{p{x),q{x)) ^- f pix)q{x)w{x){l-x)^^^\l + x)^'/^dx. (3.3) 
7^ J[-l,l] 

The notation {p,q) will refer to the inner product with respect to f{z) or w(x)(l — x"^)^^^^, whichever is 
appropriate. Thus the defining identities for these polynomials are: 

(tTu, TTm) = (5„„iV„ (3.4) 

{Pn^ ,P7n^) = SnrnN-- (3.5) 

((1 + X)p-+,p- + ) = 6„rnN-+ (3.6) 

{{l~x)p+-,p+-)=Sn,nN+- (3.7) 

((1 - X^)p++,p+ + ) = 5^raN++. (3.8) 

We also use the notation 

<(z) = z'7r,(l/z). (3.9) 
Then the following identities hold (see for example Theorem 11.5 of [^): For tt: 

m+i{z) = zm{z) + 7r,+i(0X(z) (3.10) 

13 



For p 



For p+ 



For p +: 



Ni=No n i^-^M') 

l<j<l 

77,(1)= n (i+'^jW) 

i<i<; 

(-iy7r,(-l)= n (1 + (-1)^^,(0)). 

i<j<; 



Po-((z + l/z)/2) = l 

pr(i) = 2-' n (i+'^.(o)) 

0<i<2J 

pr-(-i) = (-2)-' n (i + (-ir'r,(o)) 



o<j<2; 

0<j<2l 

= 4i/2-'iV2,(i+7r2,(0))-i 



(1 - z){2zyp+-{{z + l/z)/2) = 7r*,(z) - Z7r2;(^) 

p+-(-i) = (-2)-' n a + i-iyTTM) 

l<j<2l 

N+-=A-\l+n2i+im n (l-^i(O)') 

l<j<2l 

= 4-'iV2((l + 7r2j+i(0)) 

= 4-%,+i(l-7r2i+i(0))-^ 



(1 + z){2zypY+{{z + l/z)/2) = n;i{z) + zir^dz) 

pr+(i) = 2-' n (1+^^.(0)) 

JVr+=4-'(l-7r2m(0)) n (1-^^(0)') 

l<i<2! 

= 4-'iV2,(l-7r2(+i(0)) 
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And finally, for p++ : 

(1 - z^){2z)'p++{{z + l/z)/2) = ^2Vi(^) - ^^2i+i{z) (3.31) 

iV++=4-'-V2(i + ^2^+2(0)) n (1-^.(0)') (3-32) 

l<j<2l + l 

-4-'-i/2^2J+i(l + 7r2j+2(0)) (3.33) 

= 4-'-i/2^2/+2(l - 7r2/+2(0))-^ (3.34) 



All of the proofs are straightforward. 

We can now prove Theorem 2.3. Recall that we have chosen / to be of the form f{z) = g{z)g{\/z) such 

that the above inner product is well-defined and nondegenerate. Now consider the integral for 0+(2Z). By the 
proof of Theorem and the theory of Hankel determinants, we have 

Ec/eo+(2/)det(5(C/))cx [] N++- (3.35) 

0<j<l 

but then this is in turn proportional to 

^0 n ^2j+2(l + 7r2,+2(0))^^ (3.36) 
o<i</-i 

The constant can be determined by taking 5 = 1. The other cases are analogous. 
Theorem 3.1. Let f and g be as above. Then for any a, 

E[/GO±(2i)det((l-aC/).9(C/)) = (^2Vi(a) ± «^2i-i(a)) Eyeo±(2i) det(g((7)) (3.37) 

Ec/eo±(2i+i) det((l - aU)g{U)) = (7r*j(a) T a7r2;(a)) Ec;eo±(2;+i) det(g(?7)) (3.38) 

As a special case, 

^ueOHD dct((l + U)g{U)) = 2g{-l)-^ ^ueo-(i+i) det(g(f/)) (3.39) 

Ec/go-(odct((l + C/)g(;7)) = (3.40) 

Ec/eo+(20 det((l - U)g(U)) = 2g{l)-^ ^u^on^i+D det(5(f/)) (3.41) 

E[/GO-(20dct((l-C/)g(t/)) = (3.42) 

Ec/eo+(2i+i) det((l - U)g(U)) = (3.43) 

E[/GO-(2;+i) det((l - U)g{U)) = 2g{l)-^ ^ueo-(2i+2) det(5(C/)) (3.44) 

Proof. Follows by standard results about the behavior of orthogonal polynomials when the weight function is 

multiplied by a polynomial. □ 

Similarly, 
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Theorem 3.2. With notation as above, and for aP ^ 1, 

^ueuii) det((l - aU){l - pU^)g{U)g{U^)) = < " ^ ^^^^^^^ det(<?(t/)5(C/t)) (3.45) 
In particular, 

2g{l)g{-l){l - ap) EueuH) det((l - aU){l - (3U^)g{U)g{U^)) 

= E[/eo+(;+i) det((l - aU)g{U)) Eueo-d+i) det((l - pU)g{U)) (3.46) 
+ E^eo+(i+i) det((l - pU)g{U)) Eueo-{i+i) det((l - aU)g{U)) 

For our purposes, the following form is preferable: 

Corollary 3.3. With notation as above, and for a /3 ^ I, < 1, 

= Eyeo+(/+i) dct((l - aU){l - pU)-'g{U)) Tiueo-(i+i) det(5(C/)) (3.47) 
+ Ec/eo+((+i) det{g{U)) Eueo-(i+i) det((l - aU){l - pU)-'g{U)) 

Remark. This, together with the following theorem, enables us to compute the limits a— >±lor/3— >±1. 
Theorem 3.4. With notation as above, and for \(3\ < 1, 
Ec/eo+(2i+2) det((l - pU)-'g{U))/ Eueo+{2i+2) det{g{U)) 



2 



E[/60-(2i+2) dot((l - fiU)-'g{U))/Eueo-{2i+2) detig{U)) 

Eaeo+(2/+3) det((l - pU)-^g{U))/Eueo+{2i+3) det{g{U)) 

Ef/60-(2;+3) det((l - l3U)-'g{U))/Eueo-i2i+3) detigiU)) 

^uesp{2i) det((l - l3U)-'g{U))/EueSpi2i) det{g{U)) 
In the limits as /? — > ±1, 



/3-'{2'pr{x), (1 - /3")/(l - 2Px + P^)) 
{l-p^)N2i{l + Tr2im-^ 

(3.48) 

/3i-'(2'(l - x^)pt_\{x), 1/(1 - 2Px + p^)) 
(l-/?2)Ar2,(l-7r2i(0))-i 

(3.49) 

r'(2'(l - x)p+-{x), (1 + P)/{1 - 2px + p^)) 
(l-/32)Ar2;(l+7r2/+i(0)) 

(3.50) 

r'(2'(l + x)pi+{x), (1 - /?)/(! - 2Px + /j^)) 
(l-/32)7V2z(l-7r2;+i(0)) 

(3.51) 

P^-\2\l - x^)p++{x), 1/(1 - 2Px + /3^)) 
A^2;(l-7r2i(0))-i 

(3.52) 



lim (1 - P^) Eueo±(i+i) det((l - pU)-'g{U)) = g{l) Eueo±{i) det{g{U)) (3.53) 
^lim^(l - p') Eueo±ii+i) det((l - pU)-'g{U)) = g{-l) Ej^eo^w det{g{U)) (3.54) 
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For the symplectic group, 

lim Euespi2i) det((l - pU)-'g{U)) = gi-l)-' Eygo-(2i+i) det(<?(f/)) (3.55) 

I/3|<1 

\imEu^Spi2i) det((l - /3C/)-ig(;7)) = Ec;go+(2i+i) det(5(C/)) (3.56) 

I/3|<1 

Proof. The first group of statements are straightforward via appropriate row and column operations on the 
Hankel matrix. 

For the limits, consider 

lim (1 - (3^) Ec;go+(2i+2) det(l - (3U)-' det{g{U)) (3.57) 

I/3K1 

Here we need to compute the limit of 

r'i^'pri^). (1 " - 2/3a. + /3'))- (3.58) 

Now, 

(1 - ~ (3{z + l/z) + /32) = 1 + ^ /3"(z™ + z-"), (3.59) 

r?j.>0 

SO for any polynomial q{z), 

hm ((7(z), 1/(1 - 2/3.T + /32)) = q{l)9{l)\ (3.60) 

I/3|<1 

Plugging in and simplifying gives the desired result. The other cases are either analogous or follow from Theorem 



3.1 above. □ 



4 Diagonal points 

Recall that in defining the ensembles S® above, we insisted for and Kl that tt have no fixed points, and for S 
and Kl that 7rt have no fixed points. While these conditions are very natural from the standpoint of points in the 
square, they seem somewhat artificial in the permutation setting. This suggests consideration of the following 
extended ensembles: 

5^ = {tt e 5„ : ^ = TT-i} (4.1) 
5^ = {7re5„:^ = t^-i4 (4.2) 
5^ = {tt e S-a™ : vr = 7r-\ TT = tTT-i/,}. (4.3) 

We immediately encounter a difficulty, however, with the Poisson generating function, since the probabilities 
involve division by the complicated sum 

y {^Y^^ (4.4) 



0<m<Ln/2J 
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for S and 0, and a similar sum for Kl. However, there is a generalized Poisson model for which the generating 
function is again tractable. For 0, the model is as foUows: In a given infinitesimal time interval [t,t + dt], we 
add two points (the images of a uniform random point) with probability {l/2)tdt and add one point (a point 
uniformly chosen on the / diagonal) with probability adt. Then the probability that we have n points at time 
t is 



I 0<m< \n/2\ 



Thus if we define Pp{t;a) to be the probability that the increasing subsequence length at time t is at most 
then 

0<n ' 0<m 

where fnmi number of involutions on n elements with m fixed points and no increasing subsequence 

of length greater than I. This is, of course, compatible with our previous notation, with P^{t;Q) = PP{t). 
Similarly, for S, if we add negated points with probability (3dt, we obtain the Poisson generating function 

0<n " 0<m 

Finally, for M, we have two parameters in the model; we add fixed points with probability adt, negated points 
with probability (idt, and generic points with probability 2tdt, obtaining 

j.n 

Pf(t;a,/3)=e-"*-^*-* J] «'"+/3'"-/„V-^ (4-8) 



ni 

0<n 0<m-|-,m_ 

The P parameter is largely irrelevant, since: 
Lemma 4.1. For all I > 0, a > and /? > 0, 

Pi+i{t;0) = Pim (4.9) 
P|+i(i;a,/3)=P|(i;a,0) (4.10) 

Proof. The key observation is that if an increasing subsequence contains a given point off the \ diagonal, it 
can always be extended to contain the reflection of that point through that diagonal; moreover, no increasing 
subsequence can contain more than one point on that diagonal. Thus the point collection at time t has increasing 
subset size at most 2/ + 1 if and only if the off-diagonal subset has increasing subset size at most 21. Since the 
points were added via a generalized Poisson process, the off-diagonal subset itself corresponds to a generalized 
Poisson process; the lemma is then immediate. □ 

We can again express these generating functions as integrals: 
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Theorem 4.2. For all I, 

PP{t-a)^e-"'~'"'^^ueO{i)Act{l + aU)eMt^<U)) (4.11) 

P,^(t;l) = e-*'/2Eyeo-(i+i)exp(tTr(C/)) (4.12) 

P2l(t;/3) =e-''*-*'/2Ec;esp(2i)det(l-/3C/)-iexp(iTr(C/)) (4.13) 

Pfi+i{t\(i) = e~'"l^'&u^Spi2i) exp(tTr(C/)) (4.14) 

P,^(t;l) = e-*'/2Eyeo-(i+i)Cxp(tTr(C/)) (4.15) 

P|(t; a, /?) = e-"*-'^*-*' Ec/g[/(i) det((l + aU){l - PU)-^) exp(2i Re Tr([/)) (4.16) 

P|+i(i;a,/3) =e-"*-*'Ec;e[/(odet(l + aC/)exp(2tReTr([/)) (4.17) 

P2f+i(i; l,/3) = e-*' Ec/60-(i+2) exp(tTr(C/)) • Ec;eo-(/+i) exp(i Tr(C/)). (4.18) 

Proof. This follows immediately from the symmetric function identities of Section ^ below. (See, in particular, 



remark H to Corollary 5.5.) □ 



Remark. Strictly speaking, equations (4.13) and (4.16) are only valid when /3 < 1. The correct formulae for 
/3 > 1 are obtained by analytic continuation (e.g., by expanding det(l — PU)^^ in a formal power series, 
integrating term- by-term, and summing). 

From Theorem 4.2 and the orthogonal polynomial identities of Section ^ below, we have the following 



formulae. 

Corollary 4.3. For a,f3>0, 

Pfiit-a) = e-"*-*'/2l|[7r2Vi(-a;i) -a7r2/_i(-a;i)]i?r"W + hVi(-«;0 + a'r2/_i(-a;0]^;t+i(t)|, 

(4.19) 

P2i+i{t; a) = e-"*-*'/2 1 1 [iTli (-a; t) + a^2i (-a; t)] e*D+- {t) + [tt^^ (-a; t) - a^2i (-a; t)] e-'D-+ {t) | , 

(4.20) 

P^iit; 1) = P^iit; 1) = e-'-''/'D^+{t), (4.21) 
P^i+iit; 1) = P^i+iit; 1) = e-*^/2i^++(t), (4.22) 
Pi?+i(t;/?)=e-*V2z?++(t), (4.23) 

P2^+i(t;a,/3) = £-"*-*'<(-«; t) A (i), (4.24) 
P4^+,(i; l,/3) = e-*-*^i^++(i)i?r+W, (4.25) 
P4^+3(t; l,/3) = e-*-*^i^++(t)i?r+iW- (4.26) 

5 Schur function identities 

In order to derive the above generating functions for P'^{t;a), P^{t;P) and P^{t;a,P), we will first need a 



stronger version of Theorem 1.2 
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Remark. In the sequel, we will see a number of expressions of the form 

Eu^u^ii)det{g{U)), (5.1) 

where g takes values in a ring of formal power series (e.g., the ring of symmetric functions) with coefficients in 
C[z, 1/z] (Laurent polynomials in z). This formal integral is defined by expanding det{g{U)) and integrating 
term by term. We can recover analytical results by specializing down to a small number of variables in such a 
way that the resulting series converge. 

We refer the reader to p9| for an introduction to symmetric functions. Recall the following symmetric 
function identities: 

Lemma 5.1. The following identities hold in the algebra of symmetric functions on two sets of variables: 



And for any symmetric function f , 



It will be convenient to write 



where 



\m>0 / 



(5.3) 



{f{x),ll{l-x,y,)-')=f{y). (5.4) 



Uil ~x,yk)-' =11 H{xy,y), (5.5) 



H{u; y) [](1 - uy^y^ - ^ h,u' (5.6) 

is the generating function for the complete symmetric functions hj] by convention, hj — when j < 0. In the 
sequel, we will also need the generating function 

E{u; y) + uy,) - ^ e,u^ (5.7) 

3 3 

for the elementary symmetric functions. 
Theorem 5.2. For all I >0, 

J2 ^A(a;)sA(y) = Eueun) det{H{U; x)H{U^-y)) (5.8) 

t(X)<l 

«2A(a:) = ^ueo(i) Aet{H{U-x)) (5.9) 

1{X)<1 

J2 s^2(x) =Ec,gsp(2i)det(iI(C/;x)) (5.10) 
e{\)<2i 
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Proof. Consider the first identity. We have 

^ueu(i) det{H{U; x)H{U^;y)) = ^ sxix)s,,{y) Eueuii) sx{U)s^{U^). (5.11) 

But the integral on the right is 1 when A = /i and £(A),A(/i) < I, and is otherwise. The result follows 
immediately. 

Similarly, the identity for 0{l) follows from the fact that 

Eueoii)sx{U) (5.12) 
is 1 when A is even and £{X) < I, and is otherwise; and the identity for Sp{2l) follows from the fact that 

Eu&Sp(2i)Sx{U) (5.13) 

is 1 when A' is even and £{X) < I and is otherwise. □ 

Remark 1. This is a direct generalization of the argument in [Q. 
Remark 2. The expression det(i?([/; x)) can also be written as 

exp(V lp„(x)Tr(f/")). (5.14) 
^ — ' m 

m>0 

For the hyperoctahedral cases □ and Kl, we will need the notation 

sx{x) = s^(o) {x)sx{i) {x) (5.15) 

whenever A is a partition with trivial 2-core and 2-quotient (A*-"-* , A*-^^), and sx{x) — otherwise. From Ex. 
1.5.24 of this can also be written as 

~SX{X) = (-l)/(^)/2(/,2(s;,(x)), (5.16) 

where /(A) is the number of odd parts of A (note that this is even when A has trivial 2-core), and 02 is the 
homomorphism such that 

</'2(^2n) = hn (5.17) 

Mh2n+l) = 0, (5.18) 



or equivalently, 



Corollary 5.3. For all I > 0, 



(l)2{H(u-x)) = H{u^-x). (5.19) 



^ sx{xysx{y)^^ueu^i2i)<d^^{H{U;x)H{U^-y)) (5.20) 

l(X)<2l 

J2 S2A^(a;) =EygyH(2odet(i/([/;x)) (5.21) 

£{X)<1 
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Proof. This follows from the computations 



and 



l(X)<2l ^(A)<;, f(M)<' 

2 

Y sx{x)sx{y) 
= (Er/er/(/) dei{H{U;x)H{U^ -y)))^ 

= Ec,,,c/.ec/(0 dct(i7(C/i; x)i/(C/t; y)) Aei{H{U2;x)H{ul-v)) 
= E[;ec/(i)ec/(i) dct(i/([/;x)i7([/^;2/)) 



^ S2A2(a;)= ^ SA(a;)sA(a;) 
£(A)<; ^(A)<i 

- E[;ec/(0 det(if(C/; x)) det(if([/t; x)) 
= ^ueu^(2i) det{H{U;x)). 



(5.22) 

(5.23) 

(5.24) 
(5.25) 
(5.26) 

(5.27) 

(5.28) 
(5.29) 

□ 



The point of writing the integrands in the form det(H(U ; x)) is that we know how to convert such integrals 
into determinants: 

Theorem 5.4. For all I > 0, 



EueuH) Aei{H{U-x)H{U^-y)) 


= 'iet{gj-k{x;y))o<jM<i 


(5.30) 


^ueO+(2i)det{H{U;x)) 


= idet(ij_fc(x) + ij+fc(a;))o<j,fc<i 


(5.31) 


Ec/eo-(20det(iJ([/;x)) 


= H{l;x)H{-l;x) dct(ij_fc(a;) - ij+fe+2(a;))o<j,fc<i-i 


(5.32) 


E[/eo+(2i+i) det(iJ([/ ; x)) 


= H{l;x) det{ij-k{x) - ij+k+i{x))o<j,k<i 


(5.33) 


Ec/eo-(2i+i) det{H{U]x)) 


= H{-l;x) det{ij^k{x) + ij+k+i{x))Q<j,k<i 


(5.34) 


Ec/6Sp(20 det(H{U;x)) 


= det{ij-kix) - ij+k+2ix))o<j,k<i: 


(5.35) 



where we define 



ra-\-j (y) 

hi-^) = Yj f^m{x)hm+j{x). 



Proof. By Theorems and |2.2| , it suffices to evalute 

H{e''';x)H{e-''';y)e'''^d9. 



1 

2^ 



(5.36) 
(5.37) 

(5.38) 



[0,27r] 



22 



But this integral is just the coefficient of z ^ in x)H{z ^; y); that is, 

y^^h,n{x)hm+j{y) =5j(x;y). 

ni 

The following is then immediate: 
Corollary 5.5. For all I > 0, 

^ s\{x)sx{y) = det{gj-k{x;y))o<j,k<i 



(5.39) 
□ 



(5.40) 



l(X)<2l 



-det{ij-k{x) + ij+kix))a<j^k<i + H{1; x)H{-l] x) det{ij-kix) - ij+k+2{x))a<j,k<i-i 



/2 
(5.41) 



S2\{x) = [H{l;x)det{ij^kix) ~ ij+k+i{x))o<jM<i + H{-l;x) det{ij-k{x) + ij+k+i{x))a<j,k<i] /2 

e(\)<2i+i 



sx2{x) = det{ij-k{x) - «j+fe+2(a;))o<i,fc<i 

f(A)<2i 



(5.42) 
(5.43) 



Remark 1. The expressions for 'Ylit{X)<i^>^{^)^>^{v) ^^"^ X]f(A)<2i '^a^ (a;) are already known (||T^ and [|T6j, re- 
spectively; the latter also gives analogues of ( ^.64 ) and ( 5.65 ) below), although it is worth noting that the proof 
of the latter formula given here, in contrast to most of the proofs in the literature, involves neither pfafhans nor 



symplectic tableaux. The particular forms of ( 5.41 ) and ( 5.42 ) given here are new, although (C. Krattenthaler, 
personal communication) they can be readily derived from Theorem 2.4(3) of |3^ together with well-known 
character formulas for symplectic and special orthogonal characters; see Remark || below. The analogue of 



(5.63), again given in terms of orthogonal and symplectic characters, is given in Theorem 2 of [27 



Remark 2. We recover our earlier identities by noting that when |A| = n, (pin, s\{x)) is equal to the number 
of tableaux of shape A. So the generating functions for /® are obtained by taking inner products of the above 
formulae with exp(tpi(x)). But in general taking an inner product with the function 



exp(y] —p,-n{x)) 



(5.44) 



m>0 



gives a homomorphism from the algebra of symmetric functions to (Q)[/i, /2, . . . ], given by the substitutions 

Pm{x) >— > fm- 

In our case, we have: 



hm{x) i-> — r 

m! 

H(u;x) e"* 
i,ix)^I,i2t) 



(5.45) 

(5.46) 
(5.47) 
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so, for instance, we obtain 



J2 /"'^^ = ^ueu{i) exp(2meTr(C/)) (5.48) 

n>0 

= det(/,_fe(2i))o<,,fc<i, (5.49) 

as before. 

Remark 3. In ||3^ , it is observed that the determinant 

det(ij_fc(x) +ij-k+2{x))o<j,k<i (5.50) 

is related by duahty to a certain irreducible character of the symplectic group, and similarly the determinants 
of Corollary below are related to irreducible characters of the (odd-dimensional) special orthogonal group, 
as are those of equations ( ^.41 ) and ( 5.421 ) . It is not clear how this relates to our integrals, since the integrals 
are over groups of fixed dimension, while the characters are associated to groups of arbitrarily high dimension. 

Remark 4- The approach in for deriving such formulae is based on the operator 

/w n (5.51) 

l<j<k<l 

where (f> is the linear operator on Laurent series taking Ili<j<i IIk^x; (x). Now, this operator can be 
expressed as an integral: 

K n = / fie-') n Hie-^'^;x) Y[ (1 - e^^^e^^'') [] dO,, (5.52) 

l<j<k<l J10,2tt]' l<j<l i<j<k<l ^<j<l 

which after symmetrizing the integrand becomes simply 

Ec/6c/(i)/(f/)det(iJ(C/t;x)). (5.53) 

In |]l6| , formulae are given for a generalization of 'Y2i(\)<i ^x^i^): which the sum is over all A with a given 
number of odd parts in the dual. This is derived using the homomorphism iJ -'-(/?; a;) (||2^) which takes f{x) to 
f{f3,x) for any symmetric function /. The relevant property of H-^{(3]x) is that 

H^{l3-x)sx{x)^sx{l3.x) = ^/3l^l-l''lv(::c), (5.54) 

where /i A indicates that A — /i is a "horizontal strip"; that is, 

a: < < a: + 1 (5.55) 

for all i, or equivalently 

Ai > /zi > A2 > M2 > • ■ ■ • (5.56) 
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We will also need the dual operator E-^{a;x), which satisfies 

E^{a;x)sx{x) = ^ al^l-l^ls^(:r). (5.57) 



i<'\ 



where 



We have the identities 



M <' A -4=^ m' < A'. (5.58) 

H-^{(3; x)H{u; x) ^ (1 - l3u)-^H{u; x) (5.59) 

H^{I3] x)E{u; x)^{l + l3u)E{u; x) (5.60) 

E^{a] x)H{u; x) = (1 + au)H{u; x) (5.61) 

E'^{a;x)E{u;x) = {1 - auy^ E{u; x). (5.62) 

Theorem 5.6. For all I > 0, we have the integral identities 

J2 a^^^'^sxix) ='EueO{i)detiil + aU)H{U;x)) (5.63) 
e(\)<i 

P^^^'^s^ix) = T^uespi2i) det((l - (3U)-^H{U- x)) (5.64) 

l(X)<2l 

P^^^">sx{x)=HiP;x)i:ueSpi2i)det{H{U;x)) (5.65) 

Proof. The point is that given any partition A, there is a unique partition /i such that A <' 2/i; simply add one 
to each odd part of A, then divide by 2. In particular, then, /(A) = \2^\ — |A|, and thus 

Y a^'^^^sxix) ^ Y "'''''"'^'sa(x) (5.66) 
e{x)<i t.(jj,)<i \<'2fi 

= E^{a;x) Y «2A.(a;) (5.67) 

= Eueoii) E^{a; x) det{H{U; x)) (5.68) 

= Eueo(i) det((l + aU)HiU; x)). (5.69) 



Similarly, 



Y P^^^'hx{x)=H^{(3;x)EueSpi2i)det{H{U;x)) (5.70) 
e{\)<2i 

= Ec/65p(20 det((l - I3U)-'H{U; x)). (5.71) 



By the argument in |p6[, the expression for 



Y /3^*^'^sa(x) (5.72) 

£(A)<2i+l 
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follows from Picri's formula 

H{P;x)s^(x) = /3I^I-I^Isa(x) (5.73) 



(eq. (5.16) in Chapter I of j2^), and the fact that for every A with ^(A) < 21 + 1, there is a unique /i with 
/^^ < A, and for that ^, £{p) <l. □ 

Remark. It is worth stressing here that the integral 

^uesp(2i) det((l - I3U)-^H{U- x)) (5.74) 

is a formal integral, in which the expression (1 — Pz)^^ stands for the formal power series X]o<m We 
must therefore take special care when specializing to an analytical integral with \(3\ > 1. 



Corollary 5.7. For all I > 0, 



Proof. We have 



J2 sxix) ^ Eil;x)Eueo-ii+i)det{H{U;x)), (5.75) 

£(A)<Z 



^ sx{x) ^ det{ij_k{x) +ij+k+i{x))o<j,k<i (5.76) 

£(A)<2; 

s\{x) = H{l;x)det{ij^k{x) - ij+k+2ix))o<jM<i (5-77) 

£(A)<2; + 1 



s,ix) = Y I^^'^^a(x) (5.78) 
e(\)<i f(A)</ 

^EueOii)det{{l + U)H{U;x)) (5.79) 

= i Eueoni) dct((l + C/)i^([/; x)) (5.80) 



since det(l + C/) vanishes on O (Z). But then we can apply Theorem 3.1 to obtain 

i Eueo+(i) det((l + U)H{U; x)) = i/(-l; a;)-i E^eo- (i+i) det(iJ(C/; x)) (5.81) 
as desired (recall H{—t;x)E{t;x) — 1). The remaining formulas are immediate. □ 

Remark 1. Again, the determinantal forms are known |l| (based on an earlier pfafRan form [T5[| ). 

Remark 2. We could also have derived the formulae by the (formal) substitution /3 = 1 in the above symplectic 
formulae, or by taking the limit /3 ^ 1~. 

We also have the following formulae: 
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Theorem 5.8. For all I > 0, 

J2 a^'^^^s^ix) = E{a- x)Eu^o(i) det{H{U; x)) (5.82) 

Q,2A'i-i-/(A)^^(-^>) ^ ^(q,. x)EueO{i) det{U) det{H{U; x)). (5.83) 

\'2<1<X[ 

Proof. We recall 

E{a;x)s^{x) = ^ a^^^-^^'^ sxix); (5.84) 

that is, we sum over all partitions A obtained from fi by adjoining a vertical strip. On the other hand, 

Eu^oii)detiHiU;x))^ J2 s^i^) (5.85) 

/(A')=0 

Eu^oii)dctiU)dct{H{U;x))^ ^ s^{x). (5.86) 

£(M)=/(M)=i 

In each case, there is at most one way to remove a vertical strip from a generic partition to obtain one with 
the desired special form. Thus it remains only to determine which partitions occur. In either case, we note that 
fj,i+i = so A;+i < 1; it follows that X'2 < I. In the second case, /i; > 1, and thus A'^ > I. As these conditions 
are readily seen to be sufficient, the result follows. □ 

Remark. This gives another proof of Corollary 5/7, by subtracting (^.83 ) from ( 5.82 ) with a — 1. 

It remains to consider the formulae corresponding to hyperoctahedral involutions. For an arbitrary partition 
A, we define new partitions 

A+ = (Ai,A3,...) (5.87) 

A- = (A2,A4,...). (5.88) 

Note that A^ and A+ are the unique partitions such that (A^)'^ ^ ^ ('^^)^; ^-Iso note that /(A') = |A+| — |A^|. 

Lemma 5.9. A partition A has trivial 2-core if and only if f{X'^) = /(A^) = /(A)/2. 

Proof. A partition has trivial 2-core if and only if its diagram can be tiled by dominos. By a classical result, 
this can happen if and only if the diagram contains as many points {i,j) with i + j even as with i + j odd. For 
a, b £ {0; 1}, let Cab be the number of points in the diagram of A with (i mod 2, j mod 2) — (a. b). Then 

/(A+) = Cn - Cio (5.89) 
/(A-) = aoi-Coo. (5.90) 

But then 

/(A+) - /(A-) - Cii - Cio - Coi + Coo = 0. (5.91) 

□ 
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Theorem 5.10. For all I > 0, 

J2 af^^^^^/3^^^'^^^sx{x)=Eueu(i)<ietiil+aU)il-l3U)-^H{U;x)H{U^;x)). (5.92) 
e{\)<2i 

Proof. It follows from the lemma that for any A with trivial 2-core, there exist unique partitions n and v such 
that 

A < <' 2M^ (5.93) 

and which satisfy 

/(A) = 2/(t.) = |2MVl^'l (5.94) 

/(A') = - |A| (5.95) 

Consequently, 

h I E (-«)^'^^^/'/3^'^^'^/'sA I =</'2 [ E E^{V^;x)H^{^;x)s^^.{x)\ . (5.96) 

\e{\)<2i J \eM<i J 

It thus suffices to show that for any partition n, 

ci)2{E^{a;x)H^{h;x)s2^.{x)) = s^{x){E^{-a^;x)H^{b'';x)s^{x)), (5.97) 
since then we may set y = a; in 

E s^(y)(£^(a; x)H^{(3; x)s^{x)) = Eueu(i) det((l + aU){l - (3U)-'H{U; x)H{U^; y)). (5.98) 

e(p)<i 

By the Jacobi-Trudi identity, S2ij,2 (x) is the determinant of the block matrix 

det(S^,+j_i)i<ij<^(^), (5.99) 

where 

Brr.= (^'- (5.100) 

Equivalently, via row and column operations, we could take 

B^=[ h2^-bh2^-^ h2^_, \ ^^^^^^ 

\h2m+l - {a + b)h2m + abh2m-l h2m - ah2m-l J 

Upon applying E-^{a;x)H^{h]x) then 02, we obtain 

B' =(^"' (^ + b)H^{b^;x)hm-i \ 

" 1^0 EH-a';x)H^{b^;x)hJ' ^' ' 

But then 

det(B;^+,._Ji<i,,-<^(^) = s^{x){E^{-a^;x)H^{b^;x)s^{x)), (5.103) 

as required. □ 
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Corollary 5.11. 

Q,/(A)/2^/(A')/2g^(^) ^ ^) Ec/gc/(i) det((l + aU)H{U; x)H{U^;x)) (5.104) 

Proof. Here we use the following analogue of Pieri's formula: 

H{P- x)sx{x) = {-iy^^^'^2{H{^- x)s^{x)) (5.105) 

= ^(-l)(/W-/(A'))/2;3(|A|-|H)/2,~^(^). (5.106) 



J.>\ 



Thus 



H{p-x) a^^^hx2{x)= Y /3'^'/'"'^'a^^^^5^(^) (5.107) 



£(A)</ ^(A)<i 
A'>A= 



= Y /3l'^l/2"l^"la-^('^"^v(a;). (5.108) 
= Y /3^(^')/2a/(^)/2s^(x). (5.109) 

£(m)<2/+1 

□ 

Analogously, 
Corollary 5.12. For all I > 0, 

J2 a^<''^/^/3^(^'^/^Sp(a;) = E{a; x)Eueu{i) det((l - PU)-^H{U; x)H{U^;x)) (5.110) 

^ af^''y^pf('^"^/^S^{x) ^ E{a;x)H{p-x)Eu^u^i)H{U;x)H{U^;x)) (5.111) 

Proof. Dualizing the proof of Corollary |3.1l| , we find 

E{a;x) Y P^^^'^S2x{x) = ^af^^^'^P^^^''^'^s^{x), (5.112) 



where the sum is over ^ such that ((/i') )' = A with £(A) < I. But, as in the proof of Theorem |3.8| , this condition 
is simply that ii^ <l. The result follows. □ 



6 Pfaffian identities 

As we remarked earlier, many of the earlier proofs of the known identities from Section ^ proceeded via an 
intermediate pfaffian form. We sketch analogous proofs of the remaining identities of that section. 

We recall the definition of the pfaffian. If A is a 2n x 2n antisymmetric matrix, the pfaffian pf (A) is defined 

by: 

Pf(^)=2i' ^ ""^^^ n ^-(2j-iM2j)- (6.1) 
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In the odd-dimensional case, it will be convenient to use the notation 

pfiv;A) (6.2) 

to denote the pfaffian of A bordered by v. 

The fundamental identity we use is the following: 

Theorem 6.1. (de Bruijn, J^) Let X be a measure space, let p{x,y) be an antisymmetric function on X x X, 
and let 4>j{x) be a sequence of Junctions on X, such that for all j and k, the function (j)j{x)p{x,y)(f>k{y) is 
absolutely integrahle. Then for n even, 

'pi{p{xj,Xk))i<j,k<nAct{(j)j{xk))i<j,k<n = Ti! pf ( / (l)j{x)p{x,y)4>k{y))i<j,k<n- (6.3) 

J x,y 

For n odd, 

vi{p{xj,Xk))i<]M<n<^e^{4>j{xk))i<j,k<n =ri!pf(( / (/)j(a;))i<-,<„; ( / 4>j{x)p{x,y)4>k{y))i<j,k<n), (6.4) 

where the integrals are all over X . 
Proof. For n even, we have: 

lpf{piXj,Xk))l<j,k<ndet{(j)j{xk))l<j^k<n= ^ Cr(7ri7r2) / Y[ P(2^^l(2j-l),2:7r2(2j)) '^^(^'rsO)) 

(6.5) 

ireSn ^ l<j<n/2 l<J<n 

(6.6) 

= n!^cr(7r) Jl / (j)^{2j-i){x)p{x,y)4,^^2j){y)- (6.7) 

For n odd, we simply adjoin an extra element cxd to X and add a new function (po which is away from oo, 
then apply the identity for n even. □ 

Remark. As noted in since the proof holds for arbitrary measure spaces, this includes the case of a sum, thus 
obtaining a pfaffian analogue of the Cauchy-Binet formula (which is also readily obtained as a special case) . As 
such, this was independently rediscovered in |2l), as well as some extensions (e.g., to a g-analogue) which we 
will not need. It is also worth noting that de Bruijn's formula has applications to random matrices (pl[). 

Associated to any partition A of length < I, we associate a Z-tuple pi{X) of distinct integers, defined by 

Pi{\).j=\i-j+j, (6.8) 
with < j < L In terms of this, the Jacobi-Trudi identity becomes 

Sx{x) = det(/i^,(;^)^_j(2:))o<j,fc<;. (6.9) 
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We observe that this has the appropriate form for the application of Theorem 6.1, where we take 

= 0j(A'fe;a;) := (6.10) 

Thus, for instance, we have 

^ sx{x)sx{y) ^ ^ det((l)j(pk;x))det{(l)j{nk;y)) (6.11) 

= det(^ <Pj{f^;x)(l)k{fJ.;x))o<j.k<i (6.12) 

^ det{gj_k{x;y))o<j,k<i- (6.13) 

This is, of course, precisely Gessel's original proof (jl^), slightly restated. Similarly, the identity for □ follows 
analogously. 

To handle the involution cases, we note the following: Define a function F(/x) on Z-tuples of nonnegative 
integers as follows: if /i is increasing (and thus = /i/(A) for some A), then 

F(/i) = (6.14) 

Otherwise, F{^) is alternating under permutations of ^. 

Lemma 6.2. l2(^ If I is even, then 



F{ij) = pf{F{fj.iHk))o<i,k<i, (6.15) 

while if I is odd, then 

F{p) = - pf{F{ni)a<i^i; F{niHk)o<i,k<i)- (6.16) 

At this point, we can write 

afW/3^<^^'^sx{x)^^^ F{fi)det{cf>j{fik;x))o<j,k<i. (6.17) 

eW<i ' Mi,Ai2,...A'ieN 

For I even, this becomes: 

yj ^ pf(i^(MiMfc))o<j,fc<idet(0j(/ifc;x))o<j,fc<i = pf(^F(^t/)0j(^)0fc(z^))o<j,fc<i. (6.18) 
For I odd, we must border the pfafhan with 

(-^^^(A*)0,(M))o<j<i (6.19) 

In three cases (corresponding to 0, S, Kl), this pfaffian simplifies. We consider only in detail (in particular 
since the resulting pfaffians for the other cases are more complicated). 
For I even, after simplifying, we find 

M{j, k) := ^)<i>3i.y)M^) = k) + Afi(j, fc), (6.20) 
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where 



Mo{j,k) = ^ ^ {il + a^)i2d+i-{k-j){x)+a{i2d-{k-j){x)+i2d+2~{k-j)ix))) , (6.21) 

Q<d<{k-j) 

while Mi{j, k) ^ unless k — j is odd, when 

■ 1 - a2 

Mi(j,fc) = {-iy——H{l-x)H{~l;x) = (-l)^i?(l;a,x)i?(-l;a,x) (6.22) 

Now, Ml has rank 2, so pf (Af ) can be simplified via appropriate row and columns. To be precise, if we subtract 
row/column i from row/column j + 2 in decreasing order. Mi is now except when (j, k) — (0, 1) or (1, 0). Thus 

J2 «-''(^^sa(x) =pf(Afo) + ^ff(l;«,a;)ff(-l;a,a;)pf(M^), (6.23) 

where 

KU,k) = ik-j-i{x) - ik-j+i{x). (6.24) 
Our earlier formula follows from the following identity due to Gordon [Q: 
Theorem 6.3. Let xj be an odd function of j G Z. Then 

pf(a;fe-j)o<j,fc<2; = dct( ^ Xj^k+27n+i)a<j.k<i (6.25) 

0<m<A; 

pf((2:-'"')o<j<2i+i; (a;fc-j)o<fej<2/+i) = det( ^ (z + - (a;j_fe+2m + a;j_fc+2m+2))o<j,fc<i 

0<m<k 

(6.26) 

Remark. In addition to the proof in one can also prove this by showing that the second pfaffian gives an 
appropriate orthogonal polynomial, or by applying a special case of de Bruijn's formula (with X the unit circle). 

The computation for / odd is analogous. 



7 More increasing subsequence problems 

It turns out that the Schur function sums considered above can in many cases be given direct interpretations in 
terms of suitably defined increasing subsequences of random multisets. Let fli and Q2 be totally ordered sets, 
and let Wi C fti and W2 C ft2 be arbitrarily chosen subsets, with complements Wi and W2. It will also be 
convenient to use the corresponding indicator functions xi a-nd X2- 

Definition 2. A (Wi, W2)-increasing sequence in f^i x is a sequence {xi,yi) such that 

Xi < Xi+i, and iji < yi+i, (7.1) 

for all i, and such that 

Xi = Xi+i =^ Xi gWi, (7.2) 
Vt^Vt+i => e W2. (7.3) 
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Thus Wi specifies where the ith coordinate is allowed to weakly increase. In particular, a (Qi, f22)-iiicreasing 
sequence is weakly increasing, while a (0, 0)-increasing sequence is strictly increasing. We also have a notion of 
a {Wi,W2)-decieasmg sequence, in which the inequality for y is reversed. 

Definition 3. A finite multiset M from Oi xf22 is (Wi, W2)-compatible if for alH and j such that xi(*) 7^ X2(i), 
occurs at most once in M. 

If M is a (Wi, W2)-compatible multiset from fii x fl2, define IwiWiiM) to be the length of the longest 
(H^i, W2)-increasing subsequence of M (with, of course, the condition that an element may appear in the 
sequence at most as many times as it appears in M). We will also require the notation I^^t^^ (M) for the length 
of the longest (Wi, VF2)- decreasing subsequence of M. 

We now define, for each symmetry type, and for certain choices of (M^i, W2) for each symmetry type, a 
random (Wi, W2)-compatible multiset, and thus a random longest increasing subsequence length. We define the 
multiset by specifying for each (a;, y) the distribution of the multiplicity w{x, y) of (a;, y) in M; the multiplicities 
are independent unless otherwise specified. Each multiset distribution will also depend on certain parameters. 
For convenience, we use the following notations for the distributions which appear: g{q) for the geometric 
distribution with parameter g < 1, b{q) for a variable which is with probability 1/(1 + ?) and 1 with probability 
q/ (1 + g), and g'{oL^ q) for a variable with 

Pr(5'(a, q)=k) = ^qK (7.4) 

We denote the set of multisets from a set 5 by M(5). 

□ : The parameters are subsets W, W C Z+ and nonnegative sequences qi and q'^ such that: qtq'j < 1 whenever 
Xi(*) = X2(j) and such that 

z^,w'ii'i')-= n (^-liij) n (1 ^ 0. (7.5) 

x(i)=x'0) xWt^x'O) 
M e M(Z+ X Z+) is chosen as follows: 

X(«)=x'(j): ^ giq^qj) (7.6) 

X{t)y^x'{j)- w{i,j)r^b{q4) (7.7) 
Denote the variable lww'{M) by l^ryy/ {q; q') ■ 

□: The parameters are subsets W, W C Z symmetric under negation and nonnegative sequences qi and q^ 
such that qiq'j < 1 whenever x(«) = x'(j) and such that 

Z^w'{<1:Q')--= n (^-<l4f n (l + 94)-V0. (7.8) 

x{i)=X'{j) x{i)¥=X'{j) 

M G M(Z X Z) is chosen as follows: w{i,i) = if i = or j = 0. Otherwise: 

X{i) = X'U) ■ = w{-i,-j) ~ g{q\i\q\ji) (7.9) 

X{i) ^ X'(j) : w{i,j) = w{-i,-j) ~ 6(g|i|5[^.|) (7.10) 
Denote the variable lww'{M) by lww'il'>l')- 
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For i = j, 



0: The parameters are a subset W C Z+, a real number a > and a nonnegative sequence Qi such that 
qi <1 for all i, aqi < 1 when = 1, and 

X(»)=l X(»)=0 i<j i<j 

x{i)=xU) x(i)#x(i) 

M G M(Z+ X Z+) is chosen as follows: For i ^ j, 

X{i) = X(i) : w{i, j) = w{j,i) ~ g{qiqj) (7.12) 
X(«) 7^ X(i) : wiiJ) = wU, i) ~ h{qiqj) (7.13) 

X(«) = 1 : w{i, i) g{aqi) (7.14) 
X(i) = : w;(i,i) ~3'(a,gi). (7.15) 

Denote the variable lww{M) by /^(9; a). 

3: The parameters are a subset W C Z+, a real number /? > and a nonnegative sequence q^ such that 
q'i < 1 for all i, /^g, < 1 when x{i) = 0, and 

Z^{q;f3):= li {l~Pq^) II + U (1-*?.) H (1 + Mi)" V (7.16) 

X(j)=0 X(j) = l i<i i<J 

xb)=xU) x(»)#x(i) 

M e M(Z+ X Z") is chosen as follows: For - j, 

X{i) = x{-j) ■■ w{i, -j) = w{j, -i) ~ g{qiq-j) (7.17) 

X{i) ¥= Xi-j) ■■ w{i, -j) = w{j, -i) ~ b{qiq-j) (7.18) 

For i = — j, 

X(z) = 0: w{i,-i)^ g{l3qi) (7.19) 

X(i) = l: «;(i,-i)~5'(/3,9i)- (7-20) 

Denote the variable lw-w{M) by /^(?; /?)• 

3: The parameters are a subset W G Z symmetric imder negation, real numbers a > and /3 > 0, and a 
nonnegative sequence q^ such that qi, aq^ < 1 when x(z) = 1, q^, /3qi < 1 when x{i) = 0, and 

Z^{q-a,P):= [] (1 - /3ft)(l + "ft)"' H + Pqi^i^ - aqi)Z^w{Q;<l) i^ 0- (7-21) 

X(i)=0 xW=l 

M e M(Z X Z) is chosen as follows: w{i,j) = if i = or j = 0. Otherwise, for |i| ^ 

X{i) = X(j) : w{i,i) = w{-i,-j) = w{j,i) = w{~j,-i) ~ giq\i\q\j\) (7.22) 
X(«) 7^ X(i) : wihj) = w{-i,-j) = w{j,i) = w{-j,-i) - 6(g|i|%|); (7.23) 
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X(i) = : w{i,i) ^ w{-i,-i) ^ g'{a,q\ii) (7.24) 

X(i) = 1 : w{i,i) = w{-i,-i) ^ g{aq\,i); (7.25) 

and for i = — j, 

X(j)=0: w(i,-i) = u;(--i,i) g(/3g|,|) (7.26) 

X(i) = 1 : w{i, ~i) = w{-i, i) ^ g'{(3, g,,,). (7.27) 

Denote the variable lww{M) by l^r{q; a, (3). 

Remark. The conditions on the parameters are simply (1) that the various probability distributions arc defined, 
and (2) that M be finite with probability 1. In each case the quantity Z® gives the probability that the multiset 
is empty. 

To relate these variables to our Schur function sums, we will need the notion of a "super-Schur" (also known 
as "hook Schur") function. If Xi and yi are countable sets of variables, we define s\{x/y) to be the image of s\ 
under the homomorphism 

H{t- z) ^ H{t; x/y) := H{t; x)E{t; y). (7.28) 

(See Example 3.21 of but note that we have used a slightly different sign convention). In particular, since 
this is defined via a homomorphism, all of our identities are valid for such functions; e.g., 

sx{x/y)sx{z/w) = ^u^u[i) det{H{U; x)E{U; y)H{U^- z)E{U^-w)). (7.29) 

1{X)<1 

Then the relation to our symmetric function identities is as follows: 
Theorem 7.1. For any valid choices of parameters, 

Pr(/S'W' (9; '7')<0 = ^ww' (?;'?') E ^>^ilwl^w)sx{q!y^/(/w') (7.30) 

i(\)<i 

Pr('w'(9;9') < = ^iviv'('?;9') E ^>^i'lwl^w)sx{q^/c/w') (7-31) 

i(\)<i 

Pr(/^(g; «)</) = Z^{q- a) ^ c.^^^^ sxiq^/qw) (7.32) 

e(\)<i 

Pr{l^{q; /3) < = Z^{q-: P) E /3^'^'^^A(gvr/9w) (7.33) 

e(\)<i 

Pv{l^{q; a, /?)</) = Z^{q; a, /3) E c.^'^^^^^p^^^'^^^Sxiqw/qw)- (7.34) 

e(\)<i 

Remark. These processes are generalizations of processes studied by Johansson in p^ . In particular, he studies 
the process in the special case 

, q., = \ , (7.35) 

i> N i> M 
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as well as the process l^^ in the special case 

f V? l<i< N 

a = 1, 9, = <^ ^ . (7.36) 

[O i> N 

In both cases, he also studies an appropriate limit as g ^ 1. 

To prove this theorem, we will need a generalization of the Knuth correspondences |2^. Let be a totally 
ordered set, let be a chosen subset, and let A be a partition. 

Definition 4. An {il, VF)-bitableau T of shape A is a function from the diagram of A to which is weakly 
increasing along each row and column, such that any element of W appears at most once in each column, and 
such that any element of W appears at most once in each row. 

Remark. This is essentialy the same as the notion of bitableau given in Example 5.23 of see also [|. 

We denote the set of such bitableau as B\{Q, W), and observe (ibid.) that if Xi is a sequence of indetermi- 
nates, then for any subset W E Z+, 

^ x'^ ^ sx{xw/xj^), (7.37) 

T£Bx(Z+,W) 

where for a bitableau T, x'^ is the product of a;™' , where i appears in T rrii times. 

Theorem 7.2. Given a pair fii and ^2 of totally ordered sets, and respective subsets Wi and W2, there exists 
a bijective correspondence KwiW2 which, given a (Wi,W2)- compatible multiset M, produces a pair {P,Q) of 
bitableau of the same shape such that P is a {Q,i,Wi) -bitableau and Q is a {0,2, W2) -bitableau. A given value 
appears in the first (resp. second) tableau exactly as many times as it appears as the first (resp. second) coordinate 
in M. 

Proof. This is essentially shown in |^ ; see also . While the references only deal with the case in which every 
element of W is greater than every element of W , the proofs carry over directly. □ 

Remark 1. We have switched the order of the tableaux usually used in the Robinson-Schensted correspondence. 

Remark 2. The special cases Kq,-^^^^ and -f^Oi© are known as the Knuth correspondence and the dual Knuth 
correspondence respectively, while the special case iir00 is known as the Burge correspondence ^, [l^ . 

Since the above correspondence can be reduced to the Robinson-Schensted correspondence ([||), all of the 
usual properties carry over. We observe that the operations of inflation and deflation carry over to bitableaux, 
and thus one can define the dual P* of a bitableau. That P* is indeed a bitableau follows from the following: 

Theorem 7.3. Let ti : fii Vt'i and 1,2 '■ ^2 ^ ^'2 be order-reversing maps; also, for a multiset M , define Af* 
to be the multiset in D,2 x ^^i obtained by exchanging the coordinates. For any finite multiset AI C ili x 172, the 
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following are equivalent: 



Km w fMl = 


iP 0) 


(7 38) 




(^l(i^*),^2(Q*)) 


(7.39) 


Kw2Wi{M*) — 




(7.40) 


Kw2W^{{i2 X il)(Af*)) = 


(t2(Q*),tl(P*)) 


(7.41) 






(7.42) 


^WrT^((l X t2)(M)) = 




(7.43) 


^MWr((^2 X 1)(M*)) = 


{i2{{Q*f),P') 


(7.44) 


X .i)(M*)) = 


{Q\ii{{P*y)) 


(7.45) 



Remark. For the cases -ftrnif22 -^00 1 this theorem appears in and p3| . 

When Wi ~ W2 — W and M — AP, the two bitableau are the same; as in the Robinson-Schensted 
correspondence, we can describe the number of odd- length columns: 

Theorem 7.4. Let M be a finite multiset m il x with M — AP . If X is the common shape of Kww{M), 
then /(A') is equal to the sum of (1) the number of elements of AI of the form {x^x) with x W and (2) the 
number of x £ W that appear an odd number of times in AP 

Proof. This was known for Kn^n^ (|^^) and for ([^); it thus follows in general. □ 

Let M be a (H/i, W2)-compatible multiset from Jli x f22. We observe that lwiW2{^'^) ^ ' whenever M 
can be written as the union of / (Wi, W2)-decreasing sequences. (Again, we use {Wi, W2)-compatibility). This 
motivates the notation /[^|^^(M) for the size of the largest submultiset M' of AI with Z.Z__(M') < fc, and 



similarly for /.^^^(M). 



Theorem 7.5. Let AI be a finite (Wi,W2)- compatible multiset from fli x If ^ is the common shape of 
KwiW2{^I)r ihen 

l<i<fc l<i<fc 

Remark. For the Robinson-Schensted correspondence, this was proved in [Q; the extension to the general case 
is in §. 

In particular, Ai gives the length of the longest (Wi, W2)-increasing sequence. 



Proof of Theorem 7J_. We generalize the argument of ||2J]. Consider, for instance, the case 0. In this case, the 
probability that our random multiset AI{q] a) is equal to a given fixed multiset AI is 

Pr(M(g; a) = AI) = ¥v{M{q] a) = ^)af^^'\^, (7.47) 
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where Kww{M) = {P, P) with P of shape A. Thus 

Pr(/^(g;a) 



E 

Ai<; 



Pr(Af(g; a) = M) 



Pr{M{q; a) 



E 

i(\)<i 

PeB^,(z+,w) 



e(\)<i 



(7.48) 



(7. 



(7.50) 



Shiiilar arguments hold for types □ and S. For □ and Kl, we also need the fact that s\{x/y) can be defined 
in terms of a sum over self-dual bitableaux. Clearly, it suffices to show that the usual correspondence between 
self-dual tableaux and pairs of tableaux extends to bitableaux. 

Define a domino (il, VK)-bitableau of shape A to be a tiling of the diagram of A with labelled dominos such 
that the labels increase weakly in each row and column, and such that (1) for x G W, no column hits more 
than one domino labelled x, and (2) for x ^ W, no row hits more than one domino labelled x. We readily 
verify (using the fact that deflation preserves the bitableau property) that we have a bijection between self-dual 
bitableaux and domino bitableaux. 

To proceed from domino bitableaux to pairs of ordinary bitableaux, it suffices to show that the usual 
correspondence preserves the bitableau property. But this follows from the fact that the correspondence is valid 
for column-strict tableaux (and thus for row-strict tableaux by symmetry), as remarked in [ p7[ . 

Combining these bijections, we obtain the desired correspondence, and thus the theorem for □ and □ 



Theorem 7.1 motivates the following change of notation: 



Iww'i^l'l') 



°{qw/lw;q{j^^/q'w') 

^{qw/qv^'^qw^/qw) 
^{qw/qw\a) 

^{qw/^w,l3) 
^{qwlqw\a,i3) 



(7.51) 
(7.52) 
(7.53) 
(7.54) 
(7.55) 



It is somewhat startling that, even though the new notation in principle discards information, the resulting 
distributions are in fact exactly the same. For the involution cases, the integral formulae of Section ^ imply 
further that: 

Corollary 7.6. For any valid parameter choices, the following pairs of random variables have the same distri- 
bution: 



l^{q/r;a)^l^iq/a,r;0), 
[h^iq/r;P)\^h^iq/r;0), 
\h^iq/r■,P)^^h^iP,q/r■,0), 



(7.56) 
(7.57) 
(7.58) 
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[h^(qlr-a,P)\^P{qla,r-q/r) (7.59) 
\h^(qlr-a,P)\ ^ P{P,q/a,r-q/r) (7.60) 
where, for instance P(a,q/r;Q) corresponds to a process in which a has been inserted into the sequence q. 
Proof. We have: 

Pr{Piq/r; a) < I) = Z^{q/r; a) Eueo(i) det((l + aU)HiU; q/r)) (7.61) 
= Z^{q/a, r; 0) ^ueo[i) det(ff ((7; q/a, r)) (7.62) 

(7.63) 

and similarly for the other cases. □ 



Remark 1. By the arguments of Theorem 5.6, one can prove something stronger. Namely, the joint distribution 
of the lengths of the odd-numbered rows is the same for ^{q/r;a) and for 0(g/a, r;0), and similarly for the 
other cases. For the even-numbered rows, the argument of Theorem shows that the joint distribution 
is independent of a. In the Laguerre limit (see below), we obtain the following fact. Consider the "matrix 
ensemble" with joint eigenvalue density (on [0, oo)) proportional to 

pf(sgn(xfc -a;j)e^l^'=-="^l)i<j-fc<2„ J| {xk-Xj) ]J e-^^^ (7.64) 

l<i<fe<2ri l<j<2n 

where A and C are parameters with C > max(A, 0); note that \i xi < X2 < ■ ■ ■ X2n, then 

pf(sgn(a;fc-a;,)e^l-'=--^l)i<,,fc<2„ =exp(A ^ (-l)^a;2,). (7.65) 

l<j<2n 

Then the joint distribution of the second, fourth, sixth, etc. largest eigenvalues is independent of A. Since for 
A — this ensemble is the Laguerre orthogonal ensemble (LOE), while in the limit A — > — oo, it becomes the 
Laguerre symplectic ensemble (LSE) , we find in particular that the distribution of the second-largest eigenvalue 
of LOE is the same as the distribution of the largest eigenvalue of LSE (since every eigenvalue of LSE occurs 
twice). For an alternate proof, and generalizations, see 



Remark 2. We can recover Theorems L2 and from Theorem by taking suitable limits. For instance, 



for □, we take qi — q'i — t/N for I < i < N, and qi = q'i ~ otherwise. As iV — > oo, the resulting point 



process converges to the usual Poisson process. In Corollary [7^, if we take the corresponding limit for the 
right-hand-sides, we obtain Poisson processes in which, instead of adding extra diagonal points, we add extra 
side points. Thus we obtain the fact that these two Poisson processes have exactly the same distribution. For 
instance, if n and m are nonnegative integers, the distribution of the longest weakly increasing subsequence is 
the same if we choose either: Pick n points at random in the triangle < y < x < 1, and m points at random 
with < X — y < 1; or: Pick n points at random in the triangle < ?; < x < 1, and m points at random with 
y = and < x < 1. It is not at all clear why these distributions should be the same. 
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Remark 3. By taking a Poisson limit for only some of the variables, we obtain increasing subsequence interpre- 
tations for the image of the Schur function sums under homomorphisms of the form 

H{t;z) ^ e''*H{t;x)E{t;y) (7.66) 

with a, Xi and yi nonnegative (and satisfying the appropriate additional conditions). As this is the most general 
case in which the images of s\ are all nonnegative (||40[|), this is presumably the most general case for which 
such an interpretation can be given. 

Remark 4- In addition to the Poisson limit, another natural limit is the Laguerre limit (p^), in which the 
random multiplicities are exponentially distributed. More precisely, one fixes integers N (and N' as necessary) , 
and considers a process in which q and q' are constant on the respective intervals [1,-/V] and [l,Af'], and 
otherwise, then takes the limit q, q' — > 1. In the case a ~ P = 0, one finds the following curious fact, analogous 



to Theorem 1.2 



Given a complex N x N' matrix M, we define two decreasing sequences of nonnegative real numbers. S(M)i 
is the ith largest eigenvalue of MAff, or if i > min(7V, N'), while A(M)i is defined so that 

J2 A(M), -max ^ \Mki\^ (7.67) 
where S ranges over unions of j decreasing paths. 

Theorem 7.7. Let N be a positive even integer. Map H (recall Section ^ into transformations of N x N 
complex matrices as follows: 

/ ^ {M^ -Af*) and \ ^ (M ^ JMKJ) (7.68) 

Let G®{N) be the Gaussian distribution on matrices with the appropriate symmetry. Then for each ®, the 
distributions I](G'®(7V)) and A{G®{N)) are the same. 

Proof. (Sketch) That we can compute the distribution of A{G® (N)) (as well as the significance of this result) 
follows from the fact that it is the Laguerre limit of the appropriate discrete process with symmetry ®. Thus, 
as in [ pil for □, we find that the appropriate sum of pfaffians (see Section ^ tends to a Riemann integral. 

On the other hand, the distributions G®{N) for □, 0, and S are well-studied (these are unconstrained, 
antisymmetric, and symmetric complex matrices, respectively), and in particular the distribution Yi(G® (N)) is 
known in these cases (see |l^). For □ and we can perform simple row and column operations (not changing 
I](A'/)) to reduce to the case of □. 

We find that the two density functions we obtain are the same, proving the theorem. □ 

Remark. For □ and 0, N need not be even, while for □ and the matrices need not be square. 

8 Invariants and increasing subsequences 

Consider the integral 

Eyet/(0|Tr(C/)P". (8.1) 
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The function | Tr(?7)p" is the character of a representation of U{1), since 

|Tr(C/)|2" = Tr(C/®"0C7®"), (8.2) 

where we write A®" for the tensor product of A with itself n times. It fohows that ( ^.l[ ) (and thus f!^i) gives 
the dimension of the fixed subspace of that representation. Equivalently, this is the dimension of the space 
C„(C/(0) of operators on (C')®" that commute with f/®" for all U G [/(/). 

Given a permutation tt e 5„, we can associate an operator Ti{tt) on (C')**" as follows: 

Ti{tt){vi i»V2<» ■ ■ -Vn) ^ (w^(i) «) V^^2) <^ ■ ■ ■ W7r(n))- (8.3) 

This operator clearly commutes with J7®" for U G U{1), and thus T/ extends to a map from C[S'„] to C„(C/(Z)). 
Indeed, this map is known to be surjective (see, e.g., 0); i.e., the operators Ti{tt) span C„(C/(Z)). In general, 
however, it is not injective. For any subset S d {1,2, . . .n}, define two elements of C[S'„]: 

Es := '^W'^ (^■'^) 

7t{x)—x, Vx^S 

Hs:= (8-5) 

7r(2:)— a;, \fx^S 

Lemma 8.1. For I > n, Ti is injective on <C[Sn], while for I < n, the kernel of Ti contains all elements of the 
form ttEs with \S\ > I. 

Proof. That T/ is injective for ^ > n is straightforward; we simply observe that if wi, W2, . . . Vn are linearly 
independent vectors, then the vectors 

Tl{Tr){vi I^V2<»...Vn)= Vj,^i) (g) V^^2) <^ ■ ■ ■ Vn{n) (8.6) 

are linearly independent, as TT ranges over S^. 

Thus, suppose I < n. That ttEs E kcrT; follows from the fact that any tensor product of \S\ > I basis 
vectors must contain at least one basis vector more than once, so will be taken to by Eg- □ 



Remark. It follows from the proof of Theorem 8.2 below that these elements also span the kernel. 
Using this fact, we obtain: 

Theorem 8.2. For any nonnegative integers I and n, the set 

{Ti{7r):neSn\r{TT)<l} (8.7) 

is a basis of Cn{U{l)). 

Proof. We first need to show that, given any permutation tt with a long decreasing subsequence, we can express 
T;(7r) as a linear combination of Ti{t:') with tt' ranging over permutations without long decreasing subsequences. 
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Let TT be such a permutation, and let S* be a subset of size I + I on which tt is decreasing. By the lemma, it 
follows that 



Ti (tt) = T; (tt - ttE, 



S) [a.Q) 

= -a{n) J2 <^'W)- (8.9) 

Tt'{x)=TT(x), VX^S 
TT T^TT 

Now each permutation tt' on the right hand side agrees with tt outside S, but is no longer decreasing on S. 
It follows that each tt' has strictly fewer inversions than tt (reduce to the case in which n' differs from tt by a 
2-cycle). 

It follows that if we iterate this reduction, we will eventually obtain a linear combination of permutations 
that cannot be reduced. But this is precisely the desired result. 

It remains only to show that the elements Ti(tt) are linearly independent, which we will do via a triangularity 
argument. If we choose a basis of V, we can view T; as the restriction to {1, 2, . . . Z}" of the corresponding action 
of Sn on the set Wn = N". Now, to a given permutation tt, we associate two words 'Wi{tt) and 102 (tt) as follows: 
Wi{TT)j is equal to the length of the longest decreasing subsequence of tt starting with j; similarly W2{TT)j is equal 
to the length of the longest decreasing subsequence of tt starting in position j. (Since tt has longest decreasing 
subsequence of length < I, these are indeed in {1, 2, . . . I}"'.) We easily see that W2{tt) ~ tt{wi{tt)), and thus 
that Ti{tt) has coefficient 1 on the pair {wi{tt),'W2{tt)). The claim is then that any other permutation taking 
wi{tt) to W2{tt) has strictly more inversions than tt. 

Define the number of inversions i{'w) of a word w to be the number of coordinate positions i < j such 
that Wj < Wi. By standard arguments, we find that if tt{v) — w, then i(w) < i{v) + j(7r); equality holds 
if for any pair i < j such that tt^ > ttj, we have Vi < Vj and Wi > Wj. We immediately deduce that (a) 
i{w2(TT)) = i{wi{TT)) + i{TT) and (b) if 7r'('u;i(7r)) = ■u'2(7r), then i(7r') > i(7r). Finally, if tt' {wi{tt)) = W2{tt) with 
1(77') = i(7r), then tt' has not only the same number of inversions as tt, but indeed the same set of inversions; it 
follows that tt' — TT. □ 



Probably the most important consequence of Theorem is that it gives an elementary proof that 
Corollary 8.3. For any integers n and I, 

£;yey(,)|Tr([/)|2" = dim(C„(C/(/))) = /„;. (8.10) 

Remark 1. A reduction algorithm closely related to that used above appeared in [Q, for an application to P.I. 
algebras. The connection to invariant theory, as well as the various generalizations given below appear to be 
new, however. 

Remark 2. A different basis for I — 2 (based on one of the many combinatorial interpretations of the Catalan 
numbers) was given in 

Remark 3. We could, of course, just as easily have used the permutations without long increasing subsequences 
to form the basis. The current choice has the merit of giving a basis containing the identity and closed under 
taking inverses, as well as making the proof of linear independence somewhat cleaner. 



42 



Remark 4- While we defined everything over C, we observe that both the representation Ti and the above 
reduction algorithm are actually defined over Z. 



Remark 5. There are also, of course, analogues of Corollary 8.3 associated with Theorems 8.4 through S.8; we 
leave the details to the reader. 



More generally, Theorem 7.1 implies 

Z^W'iQ:<lT'P<lww'i<i;q') < I) = Eueuii)detiH{U;q^/qw)H{U^;qL^/q'^,)). (8.11) 
Consider the coefficient of a monomial 

U'h'il'd''^ (8-12) 

i 

in the right-hand-side. By the properties of H{t;x/y), this is 

Eueun) n ^-^(^) n ^'^^^^^ n n m^- (^-is) 

iew i(zw iew igpfF7 

Again, this is the expectation of a character, and thus computes the dimension of a space of invariants. The 
corresponding coefficient of the left-hand-side counts {W, W^')-compatible multisets without {W, M^')-increasing 
subsequences of length / + 1, in which i appears Vi times as a first coordinate and times as a second coordinate. 



Thus to obtain the analogue of Theorem B.2, we first need an analogue of Ti for multisets. 



Given a composition v, wc define a partition S{iy) of {1, 2, . . . ji'l} by 

Y^i^k}. (8.14) 

Associated to this partition is an operator 

Then the representation of U (/) in question is simply the action by conjugation of ?7®" on operators of the form 
Ti{n{i''))ATi{Il{i')). In particular, the invariant subspace is spanned by operators of the form Ti{II{i'')ttII{i')) 
with vr G Sn- We note that if two elements of Si with i E W each map to elements of the same S'j with j ^ W', 
then n(i^')7rn(z^) = 0; and similarly ii i ^ W and j G W' . It follows that operators of the form n(!^')7rn(i/) are, 
up to sign, in one-to-one correspondence with {W, VK')-compatible multisets with content {v, v'). In particular, 
given a (M^, W^')-compatible multiset M with content {v,v'), we obtain a clement of C[S'„] which we denote 
Ti{M). Writing M.^,'^,{v] v') for the set of such multisets. 

Theorem 8.4. For any nonnegative integers I and compositions v and v' , 

{Ti{M) : M e M°yy,{i^;iy')\£^^,iM) < 1} (8.16) 

IS a basis of U{v')CniU{l))U{i^). 
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Proof. Given a multiset M in Z+ x Z+, we define an inversion of AI to be a pair of elements {x,y) G M, 
{z, w) G M with X < z and y > w (i.e., a strictly decreasing subsequence of length 2). 

Now, suppose M e "-') ^ (VF, VK')-decreasing subsequence of length I + 1. Choose a per- 

mutation TT corresponding to M, and let S C {1,2, . ..n} be the set of positions of tt corresponding to the 
{W, W^')-decreasing subsequence. As before, we have 

T{Il{v')TrEsIl{v)) = 0. (8.17) 

Now, we readily see that for any permutation that appears in the left-hand-side, the corresponding multiset has 
at most as many inversions as M. Indeed, the number of inversions is strictly smaller unless the corresponding 
multiset is equal to M . Thus it remains only to show that the terms corresponding to M do not cancel. The 
only way that the term 

a{-K')n{v')-KTT'n{v) (8.18) 

can correspond to Ad is if we can write 

7r' = (n^-M.'r)(n^2J, (8.19) 

i i 

where 7r2j fixes the complement of S' n S{v)i and tt'h fixes the complement of tt{S) n S{v')i. Now, by the 
definition of {W, W^') -increasing subsequence, S n S{v)i contains at most one element when i ^ W\ similarly 
it{S) n S{v')i contains at most one element when i ^ W' . In other words, we can write 

7r' = (n (8.20) 

But then 

cr(7r')n'7r7r'n = nVn, (8.21) 

as desired. 



The proof of linear independence is analogous to that in Theorem 8.2. Of the permutations associated to M, 
there is a unique one (7r(M)) such that each element of W and W induces a decreasing subsequence and each 
element of W and W induces an increasing subsequence. We define wi(M) = wi(7r(M)), W2{M) ~ W2(7r(M)), 
and observe that any other multiset M' with a nonzero coefficient at {wi{Ad),W2{A/I)) satisfies i(7r(M')) > 
j(7r(M)). □ 

Remark 1. It is possible to renormalize the basic invariants Ti{AI) in such a way that the reduction algorithm 
is integral. We need simply divide Ti{M) by 

n \S{v\f^n{S{v'),)\\, (8.22) 

where Ti{M) = n{v')TTn{v). 
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Remark 2. The space n(j/')C„ {U (Z))n(i^) can be thought of as the space of simultaneous (multilmear) invariants 
of a coUection of symmetric and antisymmetric tensors, some covariant and some contravariant. Of special 
interest is the case in which i/' is the composition l'' , with 1, 2, . . . fc S W . In this case, the space 

{E{1.2,..,}E[i + ,,i+2,...2l} ■ ■ ■ )CniUil))U{iy) (8.23) 

corresponds to relative invariants of a collection of symmetric and antisymmetric covariant tensors; i.e., trans- 
forming the tensors multiplies the invariant by a power of the determinant. There is a known algorithm (the 
straightening algorithm [^) for computing a basis of such invariants. In fact, we observe that the resulting 
basis is, up to constant factors, the same as our basis. Thus our algorithm can be viewed as a generalization 
of this algorithm (different from the generalization to the "fourfold" algebra of [0). Similarly, the algorithms 
below for the orthogonal and symplectic groups can be thought of as straightening algorithms for those groups. 

Remark 3. There is also a "quantum" analogue of this result. If one replaces the unitary group U{1) by the 
quantum enveloping algebra Uq{Qli), the role of the symmetric group is now played by the Hecke algebra [ p2[ . 
As long as S consists of consecutive elements, there is no difficulty in defining Es and Hs (these are idempotents 
corresponding to 1-dimensional characters of parabolic subalgebras) . We find that the kernel of the quantum T; 
is the ideal generated by Es with S — {k, fc + 2, . . . fc + /}, 1 < fc < n~l. Using the appropriate normalization, we 
obtain a reduction algorithm integral over l^q]. (We also have linear independence whenever q is not a root of 
unity.) Of particular interest is the "crystal limit" q ~ Q. Under that specialization, the relations take the form 
Ti{M) — 0, with M ranging over multisets with long decreasing subsequences. This case surely merits further 
investigation, given the connection between the crystal limit of the quantum straightening algorithm and 
the Robinson-Schensted-Knuth correspondence. It would also be interesting to understand the analogues for 
the quantum orthogonal and symplectic groups. 

For the orthogonal group 0(Z), there is a "basic" invariant T^{t) associated to any fixed-point-free involution 
T in 52,1 such that the basic invariants span the invariant space of C/®^" (which we denote by F2n{0{l)). These 
transform under Ti{'k) as: 

Ti{n)T^{T)^Tf'{n-\n). (8.24) 

We write tt • t for the corresponding action on the formal span of fixed-point-free involutions. For any 
composition v and nonnegative integer a, we define 

M^iv-a) (8.25) 

to be the set of symmetric (W, VF)-compatible multisets with content composition v and with a "fixed points" 
(i.e., £ M such that i £ W, together with with i ^ W such that has odd multiplicity.) Clearly, 
there is a correspondence (up to sign) between Al^(ni; 0) and elements of the form 11 • r. 

Theorem 8.5. For any nonnegative integer I and any composition v, 

{T^{M) : M £ M^iiy; 0)| W(M) < /} (8.26) 

is a basis ofIl{i')F2„{0{l)). 
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Proof. The argument is analogous. In eliminating a given increasing subsequence, we replace both it and its 
reflection through the diagonal by non-increasing subsequences, so the number of inversions increases. 



The proof of linear independence is analogous to that of Theorem 8.4. We simply switch the reverse the 
inequalities on the second coordinates before applying the above arguments. □ 

This corresponds to taking the coefficient of a monomial q'^ in the identity 

Z^iq; a)-' Fr{l^{q; Eu^oH) det(iJ(C/; qj^/a, qw)) (8.27) 

To handle the general case, we need to map an element of M^{v; a) (corresponding to the monomial g'^a") to 
an element of 

n^(i^;a)F2„(0(/)), (8.28) 

where we define 

n^(:/; a) = n(i.)i;{|,|+i,|,|+2,...|,|+,}. (8.29) 

But this is simple: simply convert each fixed point (z, i) to a pair of elements (i, oo) and (oo, z), and apply Tf^ . 
Theorem 8.6. For any nonnegative integers I and a and any composition v , 

{Tf{M) : M e M^iv;a)\£wwiM) < 1} (8.30) 

IS a basis o/ n^(t/; a)F2„(0(/)). 

Proof. The main difficulty here is that the nai've extension of the above algorithm is no longer guaranteed to 
terminate; for instance, corresponding to the increasing subsequence 13 of 132, we have the identity 

T2^(132) = ^2^(213). (8.31) 

Since 13 is an increasing subsequence of both sides, we could clearly loop indefinitely. 

The solution is to require that the increasing subsequence consist entirely of points {x, y) with x < y. Even 
with this restriction, the above proof does not entirely carry over; for instance, both 132 and 213 have the same 
number of inversions (i.e., 1). 

For an involution r, denote by S'<(t) the set of i with i < T{i). Given a multiset M, we then define Af<(M) 
to be the multiset corresponding to S<{t) where r corresponds to M. Given two multisets Mi and M2 of the 
same size on the same totally ordered set, we write Mi < M2 to indicate that we can identify elements of Mi 
and M2 in such a way that each element of Mi is < the corresponding element of M2. Then the theorem follows 
from the following observation: 

If A/' is one of the multisets obtained after eliminating an increasing subsequence of M, then M<{M') < 
M<{M). If equality occurs, then cither M' — M or M' has strictly more inversions than M. 

Linear independence follows as above. □ 
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For the symplectic group Sp{2l), the basic invariants again correspond to involutions, but now transform 
under Ti{7r) as: 

T2i{7r)T^{T) = ±T^i{7T-'r7T), (8.32) 

with 

T2i{n)TP{T) = cT{n)TPiT) (8.33) 
whenever tt commutes with r. For any composition v and nonnegative integer b, we define 

M^,{u;b) (8.34) 
to be the set of symmetric {W, M^)-compatible multisets with content composition u and with b fixed points. 
Theorem 8.7. For any nonnegative integer I and composition v, 

{Tf (M) : M e M^,{u; 0)\l:^wiM) < 21} (8.35) 

is a basis of Il{i')F2n{Sp{2l)). 

Proof. Analogous. The only issue is that the long decreasing subsequence we eliminate must be symmetric 
about the diagonal (clearly always possible). 

For linear independence, we choose a symplectic basis of V indexed v±i, v±2. We thus find that the nonzero 
coefficients of an involution r correspond to words of length 2n on ±Z+ such that t{w) = —w. The word 
w{t) associated to an involution is now defined so that \Tj\ is equal to half the length of the longest symmetric 
decreasing subsequence starting or ending with j; the sign is positive if the sequence ends with j, and negative 
otherwise. Other than this, the arguments are analogous. □ 

This extends easily to the case when fixed points are allowed; in this case, we define 

Tlfliu; b) = n(r.)J?|,|+i,|,|+2,...|.|+6- (8-36) 

When eliminating (symmetric) decreasing subsequences, if the subsequence has even length, we can proceed 
as above; otherwise, we permute only those elements not corresponding to oo. In either case, we convert an 
decreasing subsequence to a non-decreasing subsequence. (And the linear independence argument carries over.) 
Thus 

Theorem 8.8. For any nonnegative integers I and b and any composition v, 

{Tf{M) ■ M e M'^iv-m-^w^^) < 20 (8-37) 

is a basis ofIL'^{u;b)F2n{Sp{2l)). 

It is not entirely clear how to proceed for either of □ or For or more generally U{p) x U{q), the 
centralizer algebra corresponds as expected to a certain representation Tp^g of the hyperoctahedral group, in 
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which permutations are mapped via Tp^q, and sign changes correspond to an operator with eigenvalues 1 {p 
times) and —1 (q times). Given a set S* C {1,2, . . .n}, we can define two elements Ef e C[iJ„]. Each is a 
sum over elements of iJ„ that fix the complement of S; in E'^ , we multiply by the sign of the corresponding 
permutation, while in E^ , we further multiply by the number of sign changes. Since the group U{p) x U{q) is 
a direct product, we obtain: 

Lemma 8.9. The kernel of Tp^q on C[5'ri] is spanned by the elements ttE^ with \S\ > p and by the elements 
T^Eg with \S\ > q. 

Remark. The point is that ttE^ can be used to reduce invariants of U{p), while TrEg can be used to reduce 
invariants of U{q). Since every invariant of U{p) x U{q) can be expressed in terms of the invariants of the 
respective factors, we are done. 

Even in the cases of interest, however, {q = p ot q = p + 1), it is not clear how to use these relations to 
eliminate hyperoctahedral permutations with long decreasing subsequences; in particular, the invariants do not 
span the kernel of Tp^q on Z[S'„] in general, but only on Z[1/2][S'„]. Similar remarks apply to H. As in the 
other involution cases, the invariant space for Kl is associated to a (twisted) Gelfand pair in H2n', the relevant 
subgroup of H2n is the centralizer of an element of . (For and S, the Gelfand pair is (52n, Hn), twisted in 
the S case.) 

We observe that in each case, ker(T®) C ker(T;® ^) for all I, and ker(T®) = 0. We thus obtain the following 
theorem, which is a formal analogue of the Szego limit theorem: 

Theorem 8.10. We have the following limits of formal power series: 

lim Eueuii) det{H{U; x/y)H{U^; z/w)) = 11(1 - x^Zk^^l - y^Wk)'^ Y\{1 + XjWk){l + y^zu). (8.38) 

j.k j,k 

lim Eu^oii) det{H{U;x/y)) = T\{1 + x,yu) 17(1 " x,Xk)-^ TT(1 - y^ykT^. (8.39) 

I — >oo 

j.k j<k j<k 

lim Eu^sp(2i) det{H{U;x/y)) = [](! + x,yk) - x^Xk)"' ~ y^yk)-'- (8.40) 

j,k j<k j<k 

More precisely, all coefficients of degree < 21 agree, and each limit is monotonic in all coefficients. 
Remark 1. Under the homomorphism pj(a;/?;) i-^ /i, pj{z/w) ^ gi, we obtain 

^hm Eu^uii) exp(^ /, Tr(f/^)/j + ^ g, Tr(C/-^ Vj) = exp(^ f^g^/j) (8.41) 

J J 3 

£in Eu^oii) exp(^ /, = exp{Y^{f^ + /2,)/2j) (8.42) 

3 3 

lim Euespi2i) exp(^ /, Tr(C/^")/j) - exp(^(/J - /2,)/2j) (8.43) 



and again coefficients with (weighted) degree < 21 agree. This fact was proved via representation theory in 10 



(note that the degree bounds given there are incorrect); the connection to Szego's limit theorem was observed 
in p3]. The monotonicity result is new. 
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Remark 2. There are, of course, also hyperoctahedral analogues, both of which simply reduce to the □ case. 

We can also give a partial extension of the above results to the "super" analogues of the classical groups. 
For the unitary supergroup U{l/k), the centralizer algebra Cn{U{l/k)) is again spanned by permutations, under 
a particular representation T;yj, of Sn- 

Theorem 8.11. The representations that appear in Ti/i^ with positive multiplicity are those indexed by parti- 
tions A with Xi+i < k + 1. 

We then have: 

Theorem 8.12. For any nonnegative integers I and k and compositions v and v' , the dimension of the space 
IV{v')Cn{U{l/k))Yi{v) is equal to the number of multisets M £ M^^rt{i']i'') such that 

Xww'{M)i+i <k + l. (8.44) 

Proof. If T\ is the representation of Sn corresponding to the partition A, then the dimension of 

mil{v')SnIi-{v)) (8.45) 



is given by the number of pairs of bitableau of shape A with respective content v and v' . In other words, by 



the generalized Knuth correspondence, this is equal to the number of multisets M G with 



Xww'{M) = X. (8.46) 
The result follows by summing over A. □ 
Remark 1. The obvious conjecture that those multisets give a basis under Tuj^ is not true in general (not even 



for C4(C/(1/1))); thus Theorem 3.4 does not immediately extend to the supergroup case. It is also not clear 
whether there is a simple description of the kernel of T;/^. 

Remark 2. This theorem can be thought of as a formal statement along the lines of: 

EueU{i/k)^ei{H{U-x)H{U^-y))= ^ S),{x)sx{y). (8.47) 

A, + i<fc+l 

It would be nice to make this statement precise. It is also interesting to speculate on the possibility of a "super" 
analogue of orthogonal polynomials and Toeplitz determinants. 

Similarly, for the orthosymplectic supergroup OSp{l/2k) (the full group, not just the component of the 
identity), the invariant space Fn{0 Spll / 2k)) is the image of fixed-point-free involutions under a map Tfj^^^ for 
which 

Ti,2k{^)T^2k{r) = r,%,(^-V7r). (8.48) 
This thus induces an action of S2n on F2n{0{l/2k)), for which: 
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Theorem 8.13. F2n{OSp{l/2k)) splits into Sn-irreducihle suhmodules as the direct sum 

(8.49) 



Ahn 

Aj + i<fc 



It then follows that 



Theorem 8.14. For any nonnegative integers I and k and compositions v and v' , the dimension of the space 
Il{i')F2n{OSp{l /2k)) is equal to the number of multisets M g Mww'i^'^ ^) ^'^^h that 

Xww'{M)i+i <2k + l. (8.50) 

In "integral" form, this reads: 

detiHiU;x))= ^ S2\{x). (8.51) 
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